2022-11-23T01:30:12.1887406Z Requested labels: linux.rocm.gpu 2022-11-23T01:30:12.1887480Z Job defined at: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/master 2022-11-23T01:30:12.1887502Z Waiting for a runner to pick up this job... 2022-11-23T01:30:12.3944071Z Job is about to start running on the runner: worker-rocm-amd-114 (repository) 2022-11-23T01:30:16.3888581Z Current runner version: '2.299.1' 2022-11-23T01:30:16.3895651Z Runner name: 'worker-rocm-amd-114' 2022-11-23T01:30:16.3896222Z Runner group name: 'Default' 2022-11-23T01:30:16.3896993Z Machine name: 'jenkins-worker-rocm-amd-114' 2022-11-23T01:30:16.3899516Z ##[group]GITHUB_TOKEN Permissions 2022-11-23T01:30:16.3900192Z Actions: write 2022-11-23T01:30:16.3900535Z Checks: write 2022-11-23T01:30:16.3900845Z Contents: write 2022-11-23T01:30:16.3901190Z Deployments: write 2022-11-23T01:30:16.3901541Z Discussions: write 2022-11-23T01:30:16.3901877Z Issues: write 2022-11-23T01:30:16.3902197Z Metadata: read 2022-11-23T01:30:16.3902504Z Packages: write 2022-11-23T01:30:16.3902825Z Pages: write 2022-11-23T01:30:16.3903165Z PullRequests: write 2022-11-23T01:30:16.3903558Z RepositoryProjects: write 2022-11-23T01:30:16.3903941Z SecurityEvents: write 2022-11-23T01:30:16.3904273Z Statuses: write 2022-11-23T01:30:16.3904598Z ##[endgroup] 2022-11-23T01:30:16.3908575Z Secret source: Actions 2022-11-23T01:30:16.3909249Z Prepare workflow directory 2022-11-23T01:30:16.8274792Z Prepare all required actions 2022-11-23T01:30:16.8508752Z Getting action download info 2022-11-23T01:30:17.1116570Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f) 2022-11-23T01:30:24.5929603Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd) 2022-11-23T01:30:25.8212578Z Getting action download info 2022-11-23T01:30:26.0752737Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076) 2022-11-23T01:30:27.0799566Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml 2022-11-23T01:30:27.0801784Z ##[group] Inputs 2022-11-23T01:30:27.0802199Z build-environment: linux-focal-rocm5.2-py3.8 2022-11-23T01:30:27.0802781Z test-matrix: { include: [ { config: "distributed", shard: 1, num_shards: 2, runner: "linux.rocm.gpu" }, { config: "distributed", shard: 2, num_shards: 2, runner: "linux.rocm.gpu" }, ]} 2022-11-23T01:30:27.0803508Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:30:27.0803986Z sync-tag: 2022-11-23T01:30:27.0804222Z ##[endgroup] 2022-11-23T01:30:27.0804934Z Complete job name: linux-focal-rocm5.2-py3.8-distributed / test (distributed, 1, 2, linux.rocm.gpu, rerun_disabled_tests) 2022-11-23T01:30:27.2192225Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2022-11-23T01:30:27.2192655Z with: 2022-11-23T01:30:27.2192888Z no-sudo: true 2022-11-23T01:30:27.2193166Z submodules: recursive 2022-11-23T01:30:27.2193465Z fetch-depth: 0 2022-11-23T01:30:27.2193724Z env: 2022-11-23T01:30:27.2193976Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:27.2194231Z ##[endgroup] 2022-11-23T01:30:27.2444181Z ##[group]Run retry () { 2022-11-23T01:30:27.2444453Z retry () { 2022-11-23T01:30:27.2444697Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2022-11-23T01:30:27.2444917Z } 2022-11-23T01:30:27.2445118Z echo "${GITHUB_WORKSPACE}" 2022-11-23T01:30:27.2445349Z if [ -z "${NO_SUDO}" ]; then 2022-11-23T01:30:27.2445587Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T01:30:27.2445813Z else 2022-11-23T01:30:27.2446064Z  retry rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T01:30:27.2446303Z fi 2022-11-23T01:30:27.2446498Z mkdir "${GITHUB_WORKSPACE}" 2022-11-23T01:30:27.2475714Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:30:27.2475967Z env: 2022-11-23T01:30:27.2476241Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:27.2476426Z NO_SUDO: true 2022-11-23T01:30:27.2476611Z ##[endgroup] 2022-11-23T01:30:27.2676925Z /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:29.0026622Z ##[group]Run malfet/checkout@silent-checkout 2022-11-23T01:30:29.0027051Z with: 2022-11-23T01:30:29.0027394Z ref: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:30:29.0027931Z fetch-depth: 0 2022-11-23T01:30:29.0028208Z submodules: recursive 2022-11-23T01:30:29.0028540Z quiet-checkout: true 2022-11-23T01:30:29.0028889Z repository: pytorch/pytorch 2022-11-23T01:30:29.0029525Z token: *** 2022-11-23T01:30:29.0029837Z ssh-strict: true 2022-11-23T01:30:29.0030173Z persist-credentials: true 2022-11-23T01:30:29.0030509Z clean: true 2022-11-23T01:30:29.0030802Z lfs: false 2022-11-23T01:30:29.0031124Z set-safe-directory: true 2022-11-23T01:30:29.0031444Z env: 2022-11-23T01:30:29.0031748Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:29.0032025Z ##[endgroup] 2022-11-23T01:30:29.1407807Z Syncing repository: pytorch/pytorch 2022-11-23T01:30:29.1409825Z ##[group]Getting Git version info 2022-11-23T01:30:29.1410542Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-11-23T01:30:29.1411283Z [command]/usr/bin/git version 2022-11-23T01:30:29.1411657Z git version 2.35.1 2022-11-23T01:30:29.1412555Z ##[endgroup] 2022-11-23T01:30:29.1427741Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/b9273633-4556-4a71-be79-7999456a80a3' before making global git config changes 2022-11-23T01:30:29.1428396Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T01:30:29.1429152Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:29.1429853Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-11-23T01:30:29.1430652Z ##[group]Initializing the repository 2022-11-23T01:30:29.1431390Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:29.1437374Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-11-23T01:30:29.1438060Z hint: is subject to change. To configure the initial branch name to use in all 2022-11-23T01:30:29.1438861Z hint: of your new repositories, which will suppress this warning, call: 2022-11-23T01:30:29.1439397Z hint: 2022-11-23T01:30:29.1439887Z hint: git config --global init.defaultBranch 2022-11-23T01:30:29.1440460Z hint: 2022-11-23T01:30:29.1440920Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-11-23T01:30:29.1441536Z hint: 'development'. The just-created branch can be renamed via this command: 2022-11-23T01:30:29.1442135Z hint: 2022-11-23T01:30:29.1442525Z hint: git branch -m 2022-11-23T01:30:29.1443144Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2022-11-23T01:30:29.1453335Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-11-23T01:30:29.1513648Z ##[endgroup] 2022-11-23T01:30:29.1516505Z ##[group]Disabling automatic garbage collection 2022-11-23T01:30:29.1517966Z [command]/usr/bin/git config --local gc.auto 0 2022-11-23T01:30:29.1555033Z ##[endgroup] 2022-11-23T01:30:29.1556731Z ##[group]Setting up auth 2022-11-23T01:30:29.1562149Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T01:30:29.1603796Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T01:30:29.2030068Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T01:30:29.2095812Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T01:30:29.2533006Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T01:30:29.2607743Z ##[endgroup] 2022-11-23T01:30:29.2609657Z ##[group]Fetching the repository 2022-11-23T01:30:29.2623206Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-11-23T01:31:41.9878789Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object} 2022-11-23T01:31:41.9941226Z 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:31:41.9955250Z ##[endgroup] 2022-11-23T01:31:41.9959995Z ##[group]Determining the checkout info 2022-11-23T01:31:41.9962024Z ##[endgroup] 2022-11-23T01:31:41.9963907Z ##[group]Checking out the ref 2022-11-23T01:31:41.9966980Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:31:43.2326242Z ##[endgroup] 2022-11-23T01:31:43.2328626Z ##[group]Setting up auth for fetching submodules 2022-11-23T01:31:43.2340929Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T01:31:43.2412676Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-11-23T01:31:43.2454404Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2022-11-23T01:31:43.2482612Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2022-11-23T01:31:43.2509606Z ##[endgroup] 2022-11-23T01:31:43.2511839Z ##[group]Fetching submodules 2022-11-23T01:31:43.2513761Z [command]/usr/bin/git submodule sync --recursive 2022-11-23T01:31:43.3008871Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-11-23T01:31:43.3505057Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-11-23T01:31:43.3508579Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-11-23T01:31:43.3515021Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-11-23T01:31:43.3521499Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-11-23T01:31:43.3526433Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-11-23T01:31:43.3531756Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2022-11-23T01:31:43.3537232Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-11-23T01:31:43.3541590Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-11-23T01:31:43.3545827Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-11-23T01:31:43.3549439Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-11-23T01:31:43.3553843Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-11-23T01:31:43.3557937Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2022-11-23T01:31:43.3560366Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2022-11-23T01:31:43.3562793Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-11-23T01:31:43.3565383Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-11-23T01:31:43.3567899Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-11-23T01:31:43.3571007Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-11-23T01:31:43.3575998Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:31:43.3581184Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-11-23T01:31:43.3585627Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-11-23T01:31:43.3590607Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-11-23T01:31:43.3595991Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-11-23T01:31:43.3601579Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2022-11-23T01:31:43.3607104Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-11-23T01:31:43.3612535Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-11-23T01:31:43.3618354Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-11-23T01:31:43.3624167Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2022-11-23T01:31:43.3630035Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-11-23T01:31:43.3636560Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-11-23T01:31:43.3642052Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-11-23T01:31:43.3647856Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-11-23T01:31:43.3654133Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-11-23T01:31:43.3660622Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-11-23T01:31:43.3667168Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-11-23T01:31:43.3673847Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-11-23T01:31:43.3681098Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-11-23T01:31:43.3687897Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-11-23T01:31:43.3694819Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-11-23T01:31:43.3703496Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-11-23T01:31:43.3710234Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-11-23T01:31:43.3718138Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-11-23T01:31:43.3870525Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-11-23T01:31:44.5647595Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-11-23T01:31:45.4828679Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-11-23T01:31:46.3634041Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-11-23T01:31:47.8260814Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-11-23T01:31:48.9271186Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2022-11-23T01:31:51.7507027Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-11-23T01:32:03.6460609Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-11-23T01:32:05.0604255Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-11-23T01:32:06.6312616Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-11-23T01:32:11.7090606Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-11-23T01:32:14.0599422Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2022-11-23T01:32:16.4439641Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-11-23T01:32:21.0007762Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-11-23T01:32:22.6250466Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-11-23T01:32:28.8461864Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-11-23T01:32:30.7827320Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-11-23T01:32:31.6344518Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-11-23T01:32:33.0149834Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-11-23T01:32:34.2090781Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-11-23T01:32:35.9387066Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-11-23T01:32:37.1425677Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-11-23T01:32:37.9783482Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2022-11-23T01:32:39.0210885Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-11-23T01:32:41.9736111Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-11-23T01:32:43.4206227Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-11-23T01:32:44.6269697Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2022-11-23T01:32:51.5783363Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-11-23T01:32:53.9923018Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-11-23T01:32:55.3300252Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-11-23T01:32:56.2831490Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-11-23T01:33:11.2810159Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-11-23T01:33:12.1612111Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-11-23T01:33:13.2607007Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-11-23T01:33:14.8123340Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-11-23T01:33:15.6920528Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-11-23T01:33:16.9160563Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-11-23T01:33:18.1546946Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-11-23T01:33:19.6120487Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-11-23T01:33:22.8640583Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-11-23T01:33:24.2399628Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-11-23T01:33:26.8110471Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-11-23T01:33:26.8435795Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-11-23T01:33:26.8855459Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-11-23T01:33:26.9449289Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-11-23T01:33:27.0054400Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-11-23T01:33:27.0849758Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2022-11-23T01:33:27.6664787Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-11-23T01:33:27.7281083Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:33:27.8894755Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c' 2022-11-23T01:33:27.9637830Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-11-23T01:33:28.3202348Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3' 2022-11-23T01:33:28.7118341Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a' 2022-11-23T01:33:28.9751566Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2022-11-23T01:33:29.0566764Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a' 2022-11-23T01:33:29.0643034Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:29.0646340Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:29.0652217Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:29.0658434Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:29.0758851Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-11-23T01:33:30.8783485Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-11-23T01:33:32.2883233Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-11-23T01:33:34.1250650Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2022-11-23T01:33:35.1898297Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2022-11-23T01:33:35.3500813Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-11-23T01:33:35.4678124Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-11-23T01:33:35.5128834Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2022-11-23T01:33:35.6670855Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-11-23T01:33:35.7522309Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b' 2022-11-23T01:33:35.7967219Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-11-23T01:33:35.8832546Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-11-23T01:33:35.9471401Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195' 2022-11-23T01:33:36.0401496Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-11-23T01:33:36.0896273Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f' 2022-11-23T01:33:36.0974763Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:36.1072551Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-11-23T01:33:44.3301789Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d' 2022-11-23T01:33:44.3394512Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:44.3497080Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-11-23T01:33:54.4426596Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289' 2022-11-23T01:33:54.4888696Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-11-23T01:33:54.5396849Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2022-11-23T01:33:54.6914614Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562' 2022-11-23T01:33:54.7010866Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:54.7016130Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:54.7106735Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-11-23T01:33:56.4105095Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-11-23T01:33:58.7477679Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-11-23T01:33:58.8540320Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-11-23T01:33:58.9171108Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52' 2022-11-23T01:33:58.9695741Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-11-23T01:33:59.1037161Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2022-11-23T01:33:59.3453732Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b' 2022-11-23T01:33:59.3586431Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:59.3591593Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:59.3675915Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-11-23T01:34:01.7810687Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-11-23T01:34:09.2386573Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:34:09.3139642Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8' 2022-11-23T01:34:09.3722699Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-11-23T01:34:09.3784785Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:09.3881857Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-11-23T01:34:13.8129468Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-11-23T01:34:13.8230875Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:13.8237658Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:13.8332676Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-11-23T01:34:15.0686546Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-11-23T01:34:16.8287539Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-11-23T01:34:16.9523880Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-11-23T01:34:16.9602507Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:16.9693680Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-11-23T01:34:17.9197903Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:34:17.9739851Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-11-23T01:34:18.3079136Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-11-23T01:34:18.3165181Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:18.3167429Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:18.3273385Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-11-23T01:34:19.9691109Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-11-23T01:34:22.3392506Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-11-23T01:34:22.4548520Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-11-23T01:34:22.4990740Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-11-23T01:34:22.5466826Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-11-23T01:34:22.6241751Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917' 2022-11-23T01:34:22.6664165Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-11-23T01:34:22.7392916Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2022-11-23T01:34:22.7866995Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-11-23T01:34:22.8747337Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-11-23T01:34:23.0402551Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-11-23T01:34:23.1088358Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-11-23T01:34:23.1197477Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:23.1238366Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:23.1293739Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:23.1307384Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:23.1412523Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-11-23T01:34:24.8868999Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-11-23T01:34:25.9987560Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-11-23T01:34:34.8000644Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-11-23T01:34:36.6277587Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-11-23T01:34:36.6750624Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-11-23T01:34:36.7893660Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-11-23T01:34:36.8548677Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-11-23T01:34:36.8632555Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:36.8733230Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-11-23T01:34:37.7839145Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:34:37.9998870Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-11-23T01:34:38.0169774Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-11-23T01:34:38.0693484Z Entering 'android/libs/fbjni' 2022-11-23T01:34:38.0784009Z Entering 'third_party/FP16' 2022-11-23T01:34:38.0860504Z Entering 'third_party/FXdiv' 2022-11-23T01:34:38.0944431Z Entering 'third_party/NNPACK' 2022-11-23T01:34:38.1025307Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:38.1114694Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:38.1203321Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:38.1317410Z Entering 'third_party/benchmark' 2022-11-23T01:34:38.1405768Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:38.1480678Z Entering 'third_party/cub' 2022-11-23T01:34:38.1565218Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:38.1667009Z Entering 'third_party/cutlass' 2022-11-23T01:34:38.1770028Z Entering 'third_party/eigen' 2022-11-23T01:34:38.1860247Z Entering 'third_party/fbgemm' 2022-11-23T01:34:38.1942421Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:38.2022692Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:38.2104689Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:38.2183491Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:38.2267605Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:38.2363231Z Entering 'third_party/fmt' 2022-11-23T01:34:38.2450904Z Entering 'third_party/foxi' 2022-11-23T01:34:38.2533887Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:38.2623805Z Entering 'third_party/gloo' 2022-11-23T01:34:38.2709978Z Entering 'third_party/googletest' 2022-11-23T01:34:38.2795571Z Entering 'third_party/ideep' 2022-11-23T01:34:38.2880871Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:38.2974476Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:38.3084630Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:38.3173699Z Entering 'third_party/ittapi' 2022-11-23T01:34:38.3261381Z Entering 'third_party/kineto' 2022-11-23T01:34:38.3348419Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:38.3434563Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:38.3524776Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:38.3614054Z Entering 'third_party/neon2sse' 2022-11-23T01:34:38.3703418Z Entering 'third_party/nlohmann' 2022-11-23T01:34:38.3795448Z Entering 'third_party/onnx' 2022-11-23T01:34:38.3913102Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:38.3996742Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:38.4088803Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:38.4169655Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:38.4265143Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:38.4355321Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:38.4442806Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:38.4546979Z Entering 'third_party/pocketfft' 2022-11-23T01:34:38.4637612Z Entering 'third_party/protobuf' 2022-11-23T01:34:38.4735915Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:38.4821379Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:38.4910184Z Entering 'third_party/psimd' 2022-11-23T01:34:38.4993176Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:38.5078554Z Entering 'third_party/pybind11' 2022-11-23T01:34:38.5164564Z Entering 'third_party/python-enum' 2022-11-23T01:34:38.5253271Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:38.5341317Z Entering 'third_party/python-six' 2022-11-23T01:34:38.5427646Z Entering 'third_party/sleef' 2022-11-23T01:34:38.5518205Z Entering 'third_party/tbb' 2022-11-23T01:34:38.5606982Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:38.5692924Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:38.5781651Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:38.5869082Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:38.5951764Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:38.6029002Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:38.6127147Z Entering 'third_party/zstd' 2022-11-23T01:34:38.6238947Z ##[endgroup] 2022-11-23T01:34:38.6242625Z ##[group]Persisting credentials for submodules 2022-11-23T01:34:38.6255759Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-11-23T01:34:38.6778900Z Entering 'android/libs/fbjni' 2022-11-23T01:34:38.6861207Z Entering 'third_party/FP16' 2022-11-23T01:34:38.6942490Z Entering 'third_party/FXdiv' 2022-11-23T01:34:38.7028086Z Entering 'third_party/NNPACK' 2022-11-23T01:34:38.7109856Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:38.7197532Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:38.7283200Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:38.7395140Z Entering 'third_party/benchmark' 2022-11-23T01:34:38.7478706Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:38.7562175Z Entering 'third_party/cub' 2022-11-23T01:34:38.7647309Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:38.7742985Z Entering 'third_party/cutlass' 2022-11-23T01:34:38.7849900Z Entering 'third_party/eigen' 2022-11-23T01:34:38.7934365Z Entering 'third_party/fbgemm' 2022-11-23T01:34:38.8010288Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:38.8095068Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:38.8180615Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:38.8265612Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:38.8354385Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:38.8446648Z Entering 'third_party/fmt' 2022-11-23T01:34:38.8532978Z Entering 'third_party/foxi' 2022-11-23T01:34:38.8614694Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:38.8700868Z Entering 'third_party/gloo' 2022-11-23T01:34:38.8786708Z Entering 'third_party/googletest' 2022-11-23T01:34:38.8873385Z Entering 'third_party/ideep' 2022-11-23T01:34:38.8955640Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:38.9044637Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:38.9150192Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:38.9236780Z Entering 'third_party/ittapi' 2022-11-23T01:34:38.9321485Z Entering 'third_party/kineto' 2022-11-23T01:34:38.9404808Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:38.9487977Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:38.9576378Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:38.9662268Z Entering 'third_party/neon2sse' 2022-11-23T01:34:38.9744347Z Entering 'third_party/nlohmann' 2022-11-23T01:34:38.9833367Z Entering 'third_party/onnx' 2022-11-23T01:34:38.9947926Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:39.0034683Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:39.0124211Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:39.0200362Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:39.0296625Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:39.0385005Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:39.0470260Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:39.0573082Z Entering 'third_party/pocketfft' 2022-11-23T01:34:39.0660691Z Entering 'third_party/protobuf' 2022-11-23T01:34:39.0758431Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:39.0837624Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:39.0925800Z Entering 'third_party/psimd' 2022-11-23T01:34:39.1011958Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:39.1092689Z Entering 'third_party/pybind11' 2022-11-23T01:34:39.1181402Z Entering 'third_party/python-enum' 2022-11-23T01:34:39.1268835Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:39.1350302Z Entering 'third_party/python-six' 2022-11-23T01:34:39.1434746Z Entering 'third_party/sleef' 2022-11-23T01:34:39.1520214Z Entering 'third_party/tbb' 2022-11-23T01:34:39.1612605Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:39.1700391Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:39.1786204Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:39.1870233Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:39.1950610Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:39.2033494Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:39.2127412Z Entering 'third_party/zstd' 2022-11-23T01:34:39.2244073Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-11-23T01:34:39.2773111Z Entering 'android/libs/fbjni' 2022-11-23T01:34:39.2851413Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-11-23T01:34:39.2891131Z Entering 'third_party/FP16' 2022-11-23T01:34:39.2968861Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-11-23T01:34:39.3010590Z Entering 'third_party/FXdiv' 2022-11-23T01:34:39.3087594Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-11-23T01:34:39.3127546Z Entering 'third_party/NNPACK' 2022-11-23T01:34:39.3203895Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-11-23T01:34:39.3246602Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:39.3321968Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-11-23T01:34:39.3363168Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:39.3436227Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2022-11-23T01:34:39.3478163Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:39.3551885Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-11-23T01:34:39.3619540Z Entering 'third_party/benchmark' 2022-11-23T01:34:39.3696318Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:34:39.3736974Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:39.3812969Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:34:39.3853210Z Entering 'third_party/cub' 2022-11-23T01:34:39.3928895Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-11-23T01:34:39.3971261Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:39.4049664Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-11-23T01:34:39.4103143Z Entering 'third_party/cutlass' 2022-11-23T01:34:39.4177898Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2022-11-23T01:34:39.4229046Z Entering 'third_party/eigen' 2022-11-23T01:34:39.4306226Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-11-23T01:34:39.4343111Z Entering 'third_party/fbgemm' 2022-11-23T01:34:39.4419191Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-11-23T01:34:39.4456651Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:39.4533443Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-11-23T01:34:39.4573858Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:39.4644799Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:34:39.4686628Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:39.4762698Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:34:39.4802259Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:39.4877219Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2022-11-23T01:34:39.4920241Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:39.4998050Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-11-23T01:34:39.5045637Z Entering 'third_party/fmt' 2022-11-23T01:34:39.5122840Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-11-23T01:34:39.5165182Z Entering 'third_party/foxi' 2022-11-23T01:34:39.5242157Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-11-23T01:34:39.5277238Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:39.5352377Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-11-23T01:34:39.5393766Z Entering 'third_party/gloo' 2022-11-23T01:34:39.5471397Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-11-23T01:34:39.5509662Z Entering 'third_party/googletest' 2022-11-23T01:34:39.5586593Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:34:39.5628367Z Entering 'third_party/ideep' 2022-11-23T01:34:39.5704846Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-11-23T01:34:39.5740787Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:39.5815607Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-11-23T01:34:39.5860839Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:39.5939331Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-11-23T01:34:39.6000221Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:39.6073574Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-11-23T01:34:39.6115518Z Entering 'third_party/ittapi' 2022-11-23T01:34:39.6190955Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2022-11-23T01:34:39.6229158Z Entering 'third_party/kineto' 2022-11-23T01:34:39.6306778Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-11-23T01:34:39.6345812Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:39.6420491Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-11-23T01:34:39.6463116Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:39.6533431Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-11-23T01:34:39.6573787Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:39.6650176Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-11-23T01:34:39.6693240Z Entering 'third_party/neon2sse' 2022-11-23T01:34:39.6767285Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-11-23T01:34:39.6808129Z Entering 'third_party/nlohmann' 2022-11-23T01:34:39.6886159Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2022-11-23T01:34:39.6932227Z Entering 'third_party/onnx' 2022-11-23T01:34:39.7009486Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:34:39.7086040Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:39.7164971Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:34:39.7204535Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:39.7280775Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:34:39.7327492Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:39.7403496Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-11-23T01:34:39.7443816Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:39.7519264Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:34:39.7570665Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:39.7649355Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:34:39.7690413Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:39.7768839Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:34:39.7809247Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:39.7891300Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:34:39.7950355Z Entering 'third_party/pocketfft' 2022-11-23T01:34:39.8028441Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-11-23T01:34:39.8069278Z Entering 'third_party/protobuf' 2022-11-23T01:34:39.8144292Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-11-23T01:34:39.8196670Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:39.8271024Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:34:39.8311137Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:39.8382089Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:34:39.8428710Z Entering 'third_party/psimd' 2022-11-23T01:34:39.8504977Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-11-23T01:34:39.8544747Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:39.8620346Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-11-23T01:34:39.8660030Z Entering 'third_party/pybind11' 2022-11-23T01:34:39.8735682Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:34:39.8778758Z Entering 'third_party/python-enum' 2022-11-23T01:34:39.8854906Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-11-23T01:34:39.8897268Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:39.8973415Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-11-23T01:34:39.9013602Z Entering 'third_party/python-six' 2022-11-23T01:34:39.9081860Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-11-23T01:34:39.9122775Z Entering 'third_party/sleef' 2022-11-23T01:34:39.9197909Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-11-23T01:34:39.9240475Z Entering 'third_party/tbb' 2022-11-23T01:34:39.9316605Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-11-23T01:34:39.9359413Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:39.9438268Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-11-23T01:34:39.9480251Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:39.9557227Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:34:39.9597948Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:39.9674507Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-11-23T01:34:39.9717159Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:39.9790359Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-11-23T01:34:39.9830108Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:39.9906871Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:34:39.9947099Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:40.0022852Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:34:40.0070750Z Entering 'third_party/zstd' 2022-11-23T01:34:40.0142418Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-11-23T01:34:40.0542603Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-11-23T01:34:40.1082691Z Entering 'android/libs/fbjni' 2022-11-23T01:34:40.1170580Z Entering 'third_party/FP16' 2022-11-23T01:34:40.1255788Z Entering 'third_party/FXdiv' 2022-11-23T01:34:40.1345368Z Entering 'third_party/NNPACK' 2022-11-23T01:34:40.1434214Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:40.1511545Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:40.1589701Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:40.1694514Z Entering 'third_party/benchmark' 2022-11-23T01:34:40.1779331Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:40.1864129Z Entering 'third_party/cub' 2022-11-23T01:34:40.1953382Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:40.2053950Z Entering 'third_party/cutlass' 2022-11-23T01:34:40.2158449Z Entering 'third_party/eigen' 2022-11-23T01:34:40.2254400Z Entering 'third_party/fbgemm' 2022-11-23T01:34:40.2337617Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:40.2425379Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:40.2507294Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:40.2587973Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:40.2675082Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:40.2772049Z Entering 'third_party/fmt' 2022-11-23T01:34:40.2862905Z Entering 'third_party/foxi' 2022-11-23T01:34:40.2949218Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:40.3037145Z Entering 'third_party/gloo' 2022-11-23T01:34:40.3127719Z Entering 'third_party/googletest' 2022-11-23T01:34:40.3220370Z Entering 'third_party/ideep' 2022-11-23T01:34:40.3308106Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:40.3397958Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:40.3508707Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:40.3596805Z Entering 'third_party/ittapi' 2022-11-23T01:34:40.3685730Z Entering 'third_party/kineto' 2022-11-23T01:34:40.3772713Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:40.3862035Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:40.3936436Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:40.4009503Z Entering 'third_party/neon2sse' 2022-11-23T01:34:40.4087020Z Entering 'third_party/nlohmann' 2022-11-23T01:34:40.4172363Z Entering 'third_party/onnx' 2022-11-23T01:34:40.4292230Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:40.4376741Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:40.4463097Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:40.4544712Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:40.4644693Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:40.4725932Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:40.4805080Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:40.4903782Z Entering 'third_party/pocketfft' 2022-11-23T01:34:40.4992244Z Entering 'third_party/protobuf' 2022-11-23T01:34:40.5091272Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:40.5175419Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:40.5268123Z Entering 'third_party/psimd' 2022-11-23T01:34:40.5353878Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:40.5441554Z Entering 'third_party/pybind11' 2022-11-23T01:34:40.5531698Z Entering 'third_party/python-enum' 2022-11-23T01:34:40.5618116Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:40.5706812Z Entering 'third_party/python-six' 2022-11-23T01:34:40.5794759Z Entering 'third_party/sleef' 2022-11-23T01:34:40.5884720Z Entering 'third_party/tbb' 2022-11-23T01:34:40.5976918Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:40.6063504Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:40.6151143Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:40.6238912Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:40.6329217Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:40.6399852Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:40.6475979Z Entering 'third_party/zstd' 2022-11-23T01:34:40.6589635Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2022-11-23T01:34:40.7113637Z Entering 'android/libs/fbjni' 2022-11-23T01:34:40.7198308Z Entering 'third_party/FP16' 2022-11-23T01:34:40.7288765Z Entering 'third_party/FXdiv' 2022-11-23T01:34:40.7371892Z Entering 'third_party/NNPACK' 2022-11-23T01:34:40.7461338Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:40.7552584Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:40.7642354Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:40.7756041Z Entering 'third_party/benchmark' 2022-11-23T01:34:40.7833332Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:40.7920246Z Entering 'third_party/cub' 2022-11-23T01:34:40.8003832Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:40.8107889Z Entering 'third_party/cutlass' 2022-11-23T01:34:40.8213263Z Entering 'third_party/eigen' 2022-11-23T01:34:40.8295318Z Entering 'third_party/fbgemm' 2022-11-23T01:34:40.8367013Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:40.8449469Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:40.8535546Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:40.8624525Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:40.8713143Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:40.8805288Z Entering 'third_party/fmt' 2022-11-23T01:34:40.8892318Z Entering 'third_party/foxi' 2022-11-23T01:34:40.8967686Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:40.9055576Z Entering 'third_party/gloo' 2022-11-23T01:34:40.9140018Z Entering 'third_party/googletest' 2022-11-23T01:34:40.9226842Z Entering 'third_party/ideep' 2022-11-23T01:34:40.9310794Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:40.9400902Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:40.9511783Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:40.9602535Z Entering 'third_party/ittapi' 2022-11-23T01:34:40.9687581Z Entering 'third_party/kineto' 2022-11-23T01:34:40.9772189Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:40.9856834Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:40.9945754Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:41.0032255Z Entering 'third_party/neon2sse' 2022-11-23T01:34:41.0118545Z Entering 'third_party/nlohmann' 2022-11-23T01:34:41.0195187Z Entering 'third_party/onnx' 2022-11-23T01:34:41.0306531Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:41.0392876Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:41.0486496Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:41.0571522Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:41.0668171Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:41.0757763Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:41.0847306Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:41.0951602Z Entering 'third_party/pocketfft' 2022-11-23T01:34:41.1042336Z Entering 'third_party/protobuf' 2022-11-23T01:34:41.1135823Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:41.1224130Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:41.1313228Z Entering 'third_party/psimd' 2022-11-23T01:34:41.1394914Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:41.1479178Z Entering 'third_party/pybind11' 2022-11-23T01:34:41.1563714Z Entering 'third_party/python-enum' 2022-11-23T01:34:41.1649928Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:41.1736869Z Entering 'third_party/python-six' 2022-11-23T01:34:41.1824427Z Entering 'third_party/sleef' 2022-11-23T01:34:41.1908669Z Entering 'third_party/tbb' 2022-11-23T01:34:41.1997793Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:41.2083634Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:41.2172324Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:41.2258049Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:41.2343488Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:41.2426186Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:41.2523319Z Entering 'third_party/zstd' 2022-11-23T01:34:41.2635032Z ##[endgroup] 2022-11-23T01:34:41.2738853Z [command]/usr/bin/git log -1 --format='%H' 2022-11-23T01:34:41.2809720Z '1cfd3858ac54fe3883534309081631a0a892ba3f' 2022-11-23T01:34:41.3092742Z Prepare all required actions 2022-11-23T01:34:41.3135037Z ##[group]Run ./.github/actions/setup-rocm 2022-11-23T01:34:41.3135309Z env: 2022-11-23T01:34:41.3135563Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.3135833Z ##[endgroup] 2022-11-23T01:34:41.3156860Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-11-23T01:34:41.3157347Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-11-23T01:34:41.3189848Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:41.3190165Z env: 2022-11-23T01:34:41.3190428Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.3190707Z ##[endgroup] 2022-11-23T01:34:41.3323704Z ##[group]Run cat /etc/os-release || true 2022-11-23T01:34:41.3324464Z cat /etc/os-release || true 2022-11-23T01:34:41.3325236Z cat /etc/apt/sources.list.d/rocm.list || true 2022-11-23T01:34:41.3326024Z cat /opt/rocm/.info/version || true 2022-11-23T01:34:41.3326720Z whoami 2022-11-23T01:34:41.3383111Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:41.3383840Z env: 2022-11-23T01:34:41.3384457Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.3385262Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:41.3385978Z ##[endgroup] 2022-11-23T01:34:41.3465892Z NAME="Ubuntu" 2022-11-23T01:34:41.3469925Z VERSION="18.04.5 LTS (Bionic Beaver)" 2022-11-23T01:34:41.3470741Z ID=ubuntu 2022-11-23T01:34:41.3473353Z ID_LIKE=debian 2022-11-23T01:34:41.3474079Z PRETTY_NAME="Ubuntu 18.04.5 LTS" 2022-11-23T01:34:41.3474774Z VERSION_ID="18.04" 2022-11-23T01:34:41.3479218Z HOME_URL="https://www.ubuntu.com/" 2022-11-23T01:34:41.3480202Z SUPPORT_URL="https://help.ubuntu.com/" 2022-11-23T01:34:41.3481206Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-11-23T01:34:41.3482912Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-11-23T01:34:41.3483966Z VERSION_CODENAME=bionic 2022-11-23T01:34:41.3484675Z UBUNTU_CODENAME=bionic 2022-11-23T01:34:41.3485562Z deb [arch=amd64] http://repo.radeon.com/rocm/apt/4.2 xenial main 2022-11-23T01:34:41.3501157Z 4.2.0-21 2022-11-23T01:34:41.3518950Z pytorchci 2022-11-23T01:34:41.3564163Z ##[group]Run rocm-smi 2022-11-23T01:34:41.3564826Z rocm-smi 2022-11-23T01:34:41.3609784Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:41.3610099Z env: 2022-11-23T01:34:41.3610359Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.3610708Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:41.3610988Z ##[endgroup] 2022-11-23T01:34:41.4796254Z 2022-11-23T01:34:41.4796629Z 2022-11-23T01:34:41.4797405Z ======================= ROCm System Management Interface ======================= 2022-11-23T01:34:41.4798431Z ================================= Concise Info ================================= 2022-11-23T01:34:41.4799458Z GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2022-11-23T01:34:41.4800452Z 0 29.0c 18.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:34:41.4801296Z 1 29.0c 15.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:34:41.4802151Z 2 27.0c 17.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:34:41.4802966Z 3 27.0c 17.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:34:41.4803772Z ================================================================================ 2022-11-23T01:34:41.4804625Z ============================= End of ROCm SMI Log ============================== 2022-11-23T01:34:41.4927740Z ##[group]Run rocminfo 2022-11-23T01:34:41.4928283Z rocminfo 2022-11-23T01:34:41.4982583Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:41.4983208Z env: 2022-11-23T01:34:41.4983717Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.4984389Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:41.4984965Z ##[endgroup] 2022-11-23T01:34:41.6027678Z ROCk module is loaded 2022-11-23T01:34:41.6028454Z ===================== 2022-11-23T01:34:41.6029131Z HSA System Attributes 2022-11-23T01:34:41.6029789Z ===================== 2022-11-23T01:34:41.6030439Z Runtime Version: 1.1 2022-11-23T01:34:41.6031182Z System Timestamp Freq.: 1000.000000MHz 2022-11-23T01:34:41.6032572Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2022-11-23T01:34:41.6033543Z Machine Model: LARGE 2022-11-23T01:34:41.6034468Z System Endianness: LITTLE 2022-11-23T01:34:41.6034967Z 2022-11-23T01:34:41.6035210Z ========== 2022-11-23T01:34:41.6035832Z HSA Agents 2022-11-23T01:34:41.6036762Z ========== 2022-11-23T01:34:41.6037358Z ******* 2022-11-23T01:34:41.6037951Z Agent 1 2022-11-23T01:34:41.6038517Z ******* 2022-11-23T01:34:41.6039283Z Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:34:41.6040504Z Uuid: CPU-XX 2022-11-23T01:34:41.6041397Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:34:41.6042294Z Vendor Name: CPU 2022-11-23T01:34:41.6043134Z Feature: None specified 2022-11-23T01:34:41.6043963Z Profile: FULL_PROFILE 2022-11-23T01:34:41.6045059Z Float Round Mode: NEAR 2022-11-23T01:34:41.6045879Z Max Queue Number: 0(0x0) 2022-11-23T01:34:41.6046681Z Queue Min Size: 0(0x0) 2022-11-23T01:34:41.6047461Z Queue Max Size: 0(0x0) 2022-11-23T01:34:41.6048261Z Queue Type: MULTI 2022-11-23T01:34:41.6049006Z Node: 0 2022-11-23T01:34:41.6049779Z Device Type: CPU 2022-11-23T01:34:41.6050715Z Cache Info: 2022-11-23T01:34:41.6051436Z L1: 32768(0x8000) KB 2022-11-23T01:34:41.6052205Z Chip ID: 0(0x0) 2022-11-23T01:34:41.6052980Z Cacheline Size: 64(0x40) 2022-11-23T01:34:41.6053767Z Max Clock Freq. (MHz): 3700 2022-11-23T01:34:41.6054540Z BDFID: 0 2022-11-23T01:34:41.6055323Z Internal Node ID: 0 2022-11-23T01:34:41.6056095Z Compute Unit: 28 2022-11-23T01:34:41.6056835Z SIMDs per CU: 0 2022-11-23T01:34:41.6057609Z Shader Engines: 0 2022-11-23T01:34:41.6058429Z Shader Arrs. per Eng.: 0 2022-11-23T01:34:41.6059244Z WatchPts on Addr. Ranges:1 2022-11-23T01:34:41.6059983Z Features: None 2022-11-23T01:34:41.6060641Z Pool Info: 2022-11-23T01:34:41.6061270Z Pool 1 2022-11-23T01:34:41.6062084Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:34:41.6062984Z Size: 263559840(0xfb59aa0) KB 2022-11-23T01:34:41.6063840Z Allocatable: TRUE 2022-11-23T01:34:41.6064692Z Alloc Granule: 4KB 2022-11-23T01:34:41.6065529Z Alloc Alignment: 4KB 2022-11-23T01:34:41.6066350Z Accessible by all: TRUE 2022-11-23T01:34:41.6067075Z Pool 2 2022-11-23T01:34:41.6068032Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:41.6068910Z Size: 263559840(0xfb59aa0) KB 2022-11-23T01:34:41.6069743Z Allocatable: TRUE 2022-11-23T01:34:41.6070574Z Alloc Granule: 4KB 2022-11-23T01:34:41.6071375Z Alloc Alignment: 4KB 2022-11-23T01:34:41.6072205Z Accessible by all: TRUE 2022-11-23T01:34:41.6072949Z ISA Info: 2022-11-23T01:34:41.6073583Z N/A 2022-11-23T01:34:41.6074199Z ******* 2022-11-23T01:34:41.6074799Z Agent 2 2022-11-23T01:34:41.6075381Z ******* 2022-11-23T01:34:41.6076308Z Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:34:41.6077495Z Uuid: CPU-XX 2022-11-23T01:34:41.6078382Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:34:41.6079259Z Vendor Name: CPU 2022-11-23T01:34:41.6080096Z Feature: None specified 2022-11-23T01:34:41.6080913Z Profile: FULL_PROFILE 2022-11-23T01:34:41.6081749Z Float Round Mode: NEAR 2022-11-23T01:34:41.6082556Z Max Queue Number: 0(0x0) 2022-11-23T01:34:41.6083542Z Queue Min Size: 0(0x0) 2022-11-23T01:34:41.6084331Z Queue Max Size: 0(0x0) 2022-11-23T01:34:41.6085116Z Queue Type: MULTI 2022-11-23T01:34:41.6085870Z Node: 1 2022-11-23T01:34:41.6086620Z Device Type: CPU 2022-11-23T01:34:41.6087330Z Cache Info: 2022-11-23T01:34:41.6088067Z L1: 32768(0x8000) KB 2022-11-23T01:34:41.6088839Z Chip ID: 0(0x0) 2022-11-23T01:34:41.6089624Z Cacheline Size: 64(0x40) 2022-11-23T01:34:41.6090418Z Max Clock Freq. (MHz): 3700 2022-11-23T01:34:41.6091165Z BDFID: 0 2022-11-23T01:34:41.6091926Z Internal Node ID: 1 2022-11-23T01:34:41.6092707Z Compute Unit: 28 2022-11-23T01:34:41.6093475Z SIMDs per CU: 0 2022-11-23T01:34:41.6094250Z Shader Engines: 0 2022-11-23T01:34:41.6095055Z Shader Arrs. per Eng.: 0 2022-11-23T01:34:41.6095865Z WatchPts on Addr. Ranges:1 2022-11-23T01:34:41.6096599Z Features: None 2022-11-23T01:34:41.6097258Z Pool Info: 2022-11-23T01:34:41.6097903Z Pool 1 2022-11-23T01:34:41.6098709Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:34:41.6099590Z Size: 264208872(0xfbf81e8) KB 2022-11-23T01:34:41.6100404Z Allocatable: TRUE 2022-11-23T01:34:41.6101252Z Alloc Granule: 4KB 2022-11-23T01:34:41.6102090Z Alloc Alignment: 4KB 2022-11-23T01:34:41.6102929Z Accessible by all: TRUE 2022-11-23T01:34:41.6103653Z Pool 2 2022-11-23T01:34:41.6104446Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:41.6105285Z Size: 264208872(0xfbf81e8) KB 2022-11-23T01:34:41.6106214Z Allocatable: TRUE 2022-11-23T01:34:41.6107057Z Alloc Granule: 4KB 2022-11-23T01:34:41.6107886Z Alloc Alignment: 4KB 2022-11-23T01:34:41.6108714Z Accessible by all: TRUE 2022-11-23T01:34:41.6109440Z ISA Info: 2022-11-23T01:34:41.6110087Z N/A 2022-11-23T01:34:41.6110695Z ******* 2022-11-23T01:34:41.6111305Z Agent 3 2022-11-23T01:34:41.6111903Z ******* 2022-11-23T01:34:41.6112611Z Name: gfx906 2022-11-23T01:34:41.6113724Z Uuid: GPU-17ae48c172fd5d44 2022-11-23T01:34:41.6114542Z Marketing Name: Vega 20 2022-11-23T01:34:41.6115359Z Vendor Name: AMD 2022-11-23T01:34:41.6116391Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:41.6117289Z Profile: BASE_PROFILE 2022-11-23T01:34:41.6118156Z Float Round Mode: NEAR 2022-11-23T01:34:41.6118975Z Max Queue Number: 128(0x80) 2022-11-23T01:34:41.6119767Z Queue Min Size: 4096(0x1000) 2022-11-23T01:34:41.6120558Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:41.6121508Z Queue Type: MULTI 2022-11-23T01:34:41.6122270Z Node: 2 2022-11-23T01:34:41.6123007Z Device Type: GPU 2022-11-23T01:34:41.6123703Z Cache Info: 2022-11-23T01:34:41.6124441Z L1: 16(0x10) KB 2022-11-23T01:34:41.6125236Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:41.6126034Z Cacheline Size: 64(0x40) 2022-11-23T01:34:41.6126848Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:41.6127589Z BDFID: 16128 2022-11-23T01:34:41.6128363Z Internal Node ID: 2 2022-11-23T01:34:41.6129143Z Compute Unit: 60 2022-11-23T01:34:41.6129927Z SIMDs per CU: 4 2022-11-23T01:34:41.6130716Z Shader Engines: 4 2022-11-23T01:34:41.6131526Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:41.6132356Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:41.6133146Z Features: KERNEL_DISPATCH 2022-11-23T01:34:41.6133979Z Fast F16 Operation: FALSE 2022-11-23T01:34:41.6134988Z Wavefront Size: 64(0x40) 2022-11-23T01:34:41.6135834Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:41.6136625Z Workgroup Max Size per Dimension: 2022-11-23T01:34:41.6137423Z x 1024(0x400) 2022-11-23T01:34:41.6138162Z y 1024(0x400) 2022-11-23T01:34:41.6138911Z z 1024(0x400) 2022-11-23T01:34:41.6139717Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:41.6140859Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:41.6141712Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:41.6142494Z Grid Max Size per Dimension: 2022-11-23T01:34:41.6143244Z x 4294967295(0xffffffff) 2022-11-23T01:34:41.6144172Z y 4294967295(0xffffffff) 2022-11-23T01:34:41.6144971Z z 4294967295(0xffffffff) 2022-11-23T01:34:41.6145803Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:41.6146536Z Pool Info: 2022-11-23T01:34:41.6147187Z Pool 1 2022-11-23T01:34:41.6147979Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:41.6148862Z Size: 16760832(0xffc000) KB 2022-11-23T01:34:41.6149712Z Allocatable: TRUE 2022-11-23T01:34:41.6150553Z Alloc Granule: 4KB 2022-11-23T01:34:41.6151389Z Alloc Alignment: 4KB 2022-11-23T01:34:41.6152231Z Accessible by all: FALSE 2022-11-23T01:34:41.6152938Z Pool 2 2022-11-23T01:34:41.6153698Z Segment: GROUP 2022-11-23T01:34:41.6154511Z Size: 64(0x40) KB 2022-11-23T01:34:41.6155335Z Allocatable: FALSE 2022-11-23T01:34:41.6156370Z Alloc Granule: 0KB 2022-11-23T01:34:41.6156901Z Alloc Alignment: 0KB 2022-11-23T01:34:41.6157252Z Accessible by all: FALSE 2022-11-23T01:34:41.6157612Z ISA Info: 2022-11-23T01:34:41.6157886Z ISA 1 2022-11-23T01:34:41.6158376Z Name: amdgcn-amd-amdhsa--gfx906:sramecc-:xnack- 2022-11-23T01:34:41.6158789Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:41.6159163Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:41.6159531Z Default Rounding Mode: NEAR 2022-11-23T01:34:41.6159869Z Default Rounding Mode: NEAR 2022-11-23T01:34:41.6160212Z Fast f16: TRUE 2022-11-23T01:34:41.6160558Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:41.6160893Z Workgroup Max Size per Dimension: 2022-11-23T01:34:41.6161228Z x 1024(0x400) 2022-11-23T01:34:41.6161549Z y 1024(0x400) 2022-11-23T01:34:41.6161847Z z 1024(0x400) 2022-11-23T01:34:41.6162175Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:41.6162495Z Grid Max Size per Dimension: 2022-11-23T01:34:41.6162819Z x 4294967295(0xffffffff) 2022-11-23T01:34:41.6163151Z y 4294967295(0xffffffff) 2022-11-23T01:34:41.6163491Z z 4294967295(0xffffffff) 2022-11-23T01:34:41.6163807Z FBarrier Max Size: 32 2022-11-23T01:34:41.6164101Z ******* 2022-11-23T01:34:41.6164360Z Agent 4 2022-11-23T01:34:41.6164619Z ******* 2022-11-23T01:34:41.6164910Z Name: gfx906 2022-11-23T01:34:41.6165357Z Uuid: GPU-17dc608172fd5d44 2022-11-23T01:34:41.6165687Z Marketing Name: Vega 20 2022-11-23T01:34:41.6166022Z Vendor Name: AMD 2022-11-23T01:34:41.6166364Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:41.6166717Z Profile: BASE_PROFILE 2022-11-23T01:34:41.6167061Z Float Round Mode: NEAR 2022-11-23T01:34:41.6167466Z Max Queue Number: 128(0x80) 2022-11-23T01:34:41.6167804Z Queue Min Size: 4096(0x1000) 2022-11-23T01:34:41.6168113Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:41.6168441Z Queue Type: MULTI 2022-11-23T01:34:41.6168756Z Node: 3 2022-11-23T01:34:41.6169073Z Device Type: GPU 2022-11-23T01:34:41.6169375Z Cache Info: 2022-11-23T01:34:41.6169679Z L1: 16(0x10) KB 2022-11-23T01:34:41.6169984Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:41.6170414Z Cacheline Size: 64(0x40) 2022-11-23T01:34:41.6170738Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:41.6171041Z BDFID: 17152 2022-11-23T01:34:41.6171363Z Internal Node ID: 3 2022-11-23T01:34:41.6171684Z Compute Unit: 60 2022-11-23T01:34:41.6172007Z SIMDs per CU: 4 2022-11-23T01:34:41.6172327Z Shader Engines: 4 2022-11-23T01:34:41.6172666Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:41.6173069Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:41.6173390Z Features: KERNEL_DISPATCH 2022-11-23T01:34:41.6173864Z Fast F16 Operation: FALSE 2022-11-23T01:34:41.6174209Z Wavefront Size: 64(0x40) 2022-11-23T01:34:41.6174552Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:41.6174875Z Workgroup Max Size per Dimension: 2022-11-23T01:34:41.6175204Z x 1024(0x400) 2022-11-23T01:34:41.6175520Z y 1024(0x400) 2022-11-23T01:34:41.6175805Z z 1024(0x400) 2022-11-23T01:34:41.6176123Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:41.6176577Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:41.6176930Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:41.6177248Z Grid Max Size per Dimension: 2022-11-23T01:34:41.6177563Z x 4294967295(0xffffffff) 2022-11-23T01:34:41.6177869Z y 4294967295(0xffffffff) 2022-11-23T01:34:41.6178186Z z 4294967295(0xffffffff) 2022-11-23T01:34:41.6178577Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:41.6178881Z Pool Info: 2022-11-23T01:34:41.6179151Z Pool 1 2022-11-23T01:34:41.6179486Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:41.6179827Z Size: 16760832(0xffc000) KB 2022-11-23T01:34:41.6180171Z Allocatable: TRUE 2022-11-23T01:34:41.6180521Z Alloc Granule: 4KB 2022-11-23T01:34:41.6180860Z Alloc Alignment: 4KB 2022-11-23T01:34:41.6181219Z Accessible by all: FALSE 2022-11-23T01:34:41.6181527Z Pool 2 2022-11-23T01:34:41.6181816Z Segment: GROUP 2022-11-23T01:34:41.6182150Z Size: 64(0x40) KB 2022-11-23T01:34:41.6182483Z Allocatable: FALSE 2022-11-23T01:34:41.6182884Z Alloc Granule: 0KB 2022-11-23T01:34:41.6183226Z Alloc Alignment: 0KB 2022-11-23T01:34:41.6183568Z Accessible by all: FALSE 2022-11-23T01:34:41.6183869Z ISA Info: 2022-11-23T01:34:41.6184120Z ISA 1 2022-11-23T01:34:41.6184586Z Name: amdgcn-amd-amdhsa--gfx906:sramecc-:xnack- 2022-11-23T01:34:41.6184990Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:41.6185364Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:41.6185721Z Default Rounding Mode: NEAR 2022-11-23T01:34:41.6186073Z Default Rounding Mode: NEAR 2022-11-23T01:34:41.6186398Z Fast f16: TRUE 2022-11-23T01:34:41.6186739Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:41.6187077Z Workgroup Max Size per Dimension: 2022-11-23T01:34:41.6187408Z x 1024(0x400) 2022-11-23T01:34:41.6187722Z y 1024(0x400) 2022-11-23T01:34:41.6188036Z z 1024(0x400) 2022-11-23T01:34:41.6188344Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:41.6188666Z Grid Max Size per Dimension: 2022-11-23T01:34:41.6189080Z x 4294967295(0xffffffff) 2022-11-23T01:34:41.6189412Z y 4294967295(0xffffffff) 2022-11-23T01:34:41.6189747Z z 4294967295(0xffffffff) 2022-11-23T01:34:41.6190082Z FBarrier Max Size: 32 2022-11-23T01:34:41.6190355Z ******* 2022-11-23T01:34:41.6190613Z Agent 5 2022-11-23T01:34:41.6190864Z ******* 2022-11-23T01:34:41.6191152Z Name: gfx906 2022-11-23T01:34:41.6191592Z Uuid: GPU-17a8612172fd5d44 2022-11-23T01:34:41.6191929Z Marketing Name: Vega 20 2022-11-23T01:34:41.6192241Z Vendor Name: AMD 2022-11-23T01:34:41.6192578Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:41.6192927Z Profile: BASE_PROFILE 2022-11-23T01:34:41.6193262Z Float Round Mode: NEAR 2022-11-23T01:34:41.6193591Z Max Queue Number: 128(0x80) 2022-11-23T01:34:41.6193921Z Queue Min Size: 4096(0x1000) 2022-11-23T01:34:41.6194242Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:41.6194553Z Queue Type: MULTI 2022-11-23T01:34:41.6194866Z Node: 4 2022-11-23T01:34:41.6195177Z Device Type: GPU 2022-11-23T01:34:41.6195470Z Cache Info: 2022-11-23T01:34:41.6195838Z L1: 16(0x10) KB 2022-11-23T01:34:41.6196295Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:41.6196655Z Cacheline Size: 64(0x40) 2022-11-23T01:34:41.6197069Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:41.6197471Z BDFID: 17920 2022-11-23T01:34:41.6197942Z Internal Node ID: 4 2022-11-23T01:34:41.6198372Z Compute Unit: 60 2022-11-23T01:34:41.6198853Z SIMDs per CU: 4 2022-11-23T01:34:41.6199263Z Shader Engines: 4 2022-11-23T01:34:41.6199731Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:41.6234434Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:41.6235027Z Features: KERNEL_DISPATCH 2022-11-23T01:34:41.6235408Z Fast F16 Operation: FALSE 2022-11-23T01:34:41.6235783Z Wavefront Size: 64(0x40) 2022-11-23T01:34:41.6236285Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:41.6236639Z Workgroup Max Size per Dimension: 2022-11-23T01:34:41.6236958Z x 1024(0x400) 2022-11-23T01:34:41.6237288Z y 1024(0x400) 2022-11-23T01:34:41.6237617Z z 1024(0x400) 2022-11-23T01:34:41.6237966Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:41.6238537Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:41.6238908Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:41.6239222Z Grid Max Size per Dimension: 2022-11-23T01:34:41.6239553Z x 4294967295(0xffffffff) 2022-11-23T01:34:41.6239897Z y 4294967295(0xffffffff) 2022-11-23T01:34:41.6240464Z z 4294967295(0xffffffff) 2022-11-23T01:34:41.6240814Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:41.6241128Z Pool Info: 2022-11-23T01:34:41.6241394Z Pool 1 2022-11-23T01:34:41.6241780Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:41.6242177Z Size: 16760832(0xffc000) KB 2022-11-23T01:34:41.6242538Z Allocatable: TRUE 2022-11-23T01:34:41.6242888Z Alloc Granule: 4KB 2022-11-23T01:34:41.6243228Z Alloc Alignment: 4KB 2022-11-23T01:34:41.6243566Z Accessible by all: FALSE 2022-11-23T01:34:41.6243882Z Pool 2 2022-11-23T01:34:41.6244196Z Segment: GROUP 2022-11-23T01:34:41.6244539Z Size: 64(0x40) KB 2022-11-23T01:34:41.6244888Z Allocatable: FALSE 2022-11-23T01:34:41.6245239Z Alloc Granule: 0KB 2022-11-23T01:34:41.6245564Z Alloc Alignment: 0KB 2022-11-23T01:34:41.6245918Z Accessible by all: FALSE 2022-11-23T01:34:41.6246236Z ISA Info: 2022-11-23T01:34:41.6246518Z ISA 1 2022-11-23T01:34:41.6247029Z Name: amdgcn-amd-amdhsa--gfx906:sramecc-:xnack- 2022-11-23T01:34:41.6247454Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:41.6247839Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:41.6248192Z Default Rounding Mode: NEAR 2022-11-23T01:34:41.6248566Z Default Rounding Mode: NEAR 2022-11-23T01:34:41.6248920Z Fast f16: TRUE 2022-11-23T01:34:41.6249265Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:41.6249606Z Workgroup Max Size per Dimension: 2022-11-23T01:34:41.6249945Z x 1024(0x400) 2022-11-23T01:34:41.6250255Z y 1024(0x400) 2022-11-23T01:34:41.6250670Z z 1024(0x400) 2022-11-23T01:34:41.6251018Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:41.6251357Z Grid Max Size per Dimension: 2022-11-23T01:34:41.6251706Z x 4294967295(0xffffffff) 2022-11-23T01:34:41.6252059Z y 4294967295(0xffffffff) 2022-11-23T01:34:41.6252382Z z 4294967295(0xffffffff) 2022-11-23T01:34:41.6252726Z FBarrier Max Size: 32 2022-11-23T01:34:41.6253032Z ******* 2022-11-23T01:34:41.6253292Z Agent 6 2022-11-23T01:34:41.6253551Z ******* 2022-11-23T01:34:41.6253853Z Name: gfx906 2022-11-23T01:34:41.6254297Z Uuid: GPU-d422708172dc76e6 2022-11-23T01:34:41.6254662Z Marketing Name: Vega 20 2022-11-23T01:34:41.6255010Z Vendor Name: AMD 2022-11-23T01:34:41.6255356Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:41.6255715Z Profile: BASE_PROFILE 2022-11-23T01:34:41.6256069Z Float Round Mode: NEAR 2022-11-23T01:34:41.6256396Z Max Queue Number: 128(0x80) 2022-11-23T01:34:41.6256811Z Queue Min Size: 4096(0x1000) 2022-11-23T01:34:41.6257147Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:41.6257476Z Queue Type: MULTI 2022-11-23T01:34:41.6257803Z Node: 5 2022-11-23T01:34:41.6258139Z Device Type: GPU 2022-11-23T01:34:41.6258445Z Cache Info: 2022-11-23T01:34:41.6258743Z L1: 16(0x10) KB 2022-11-23T01:34:41.6259079Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:41.6259417Z Cacheline Size: 64(0x40) 2022-11-23T01:34:41.6259760Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:41.6260091Z BDFID: 18688 2022-11-23T01:34:41.6260423Z Internal Node ID: 5 2022-11-23T01:34:41.6260745Z Compute Unit: 60 2022-11-23T01:34:41.6282106Z SIMDs per CU: 4 2022-11-23T01:34:41.6282599Z Shader Engines: 4 2022-11-23T01:34:41.6282973Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:41.6283343Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:41.6283718Z Features: KERNEL_DISPATCH 2022-11-23T01:34:41.6284072Z Fast F16 Operation: FALSE 2022-11-23T01:34:41.6284440Z Wavefront Size: 64(0x40) 2022-11-23T01:34:41.6284841Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:41.6285185Z Workgroup Max Size per Dimension: 2022-11-23T01:34:41.6285526Z x 1024(0x400) 2022-11-23T01:34:41.6285864Z y 1024(0x400) 2022-11-23T01:34:41.6286188Z z 1024(0x400) 2022-11-23T01:34:41.6286508Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:41.6287070Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:41.6287438Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:41.6287989Z Grid Max Size per Dimension: 2022-11-23T01:34:41.6288333Z x 4294967295(0xffffffff) 2022-11-23T01:34:41.6288671Z y 4294967295(0xffffffff) 2022-11-23T01:34:41.6288989Z z 4294967295(0xffffffff) 2022-11-23T01:34:41.6289333Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:41.6289651Z Pool Info: 2022-11-23T01:34:41.6289934Z Pool 1 2022-11-23T01:34:41.6290285Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:41.6290655Z Size: 16760832(0xffc000) KB 2022-11-23T01:34:41.6290987Z Allocatable: TRUE 2022-11-23T01:34:41.6291343Z Alloc Granule: 4KB 2022-11-23T01:34:41.6291691Z Alloc Alignment: 4KB 2022-11-23T01:34:41.6292056Z Accessible by all: FALSE 2022-11-23T01:34:41.6292369Z Pool 2 2022-11-23T01:34:41.6292695Z Segment: GROUP 2022-11-23T01:34:41.6293015Z Size: 64(0x40) KB 2022-11-23T01:34:41.6293357Z Allocatable: FALSE 2022-11-23T01:34:41.6293712Z Alloc Granule: 0KB 2022-11-23T01:34:41.6294152Z Alloc Alignment: 0KB 2022-11-23T01:34:41.6294512Z Accessible by all: FALSE 2022-11-23T01:34:41.6294826Z ISA Info: 2022-11-23T01:34:41.6295086Z ISA 1 2022-11-23T01:34:41.6295593Z Name: amdgcn-amd-amdhsa--gfx906:sramecc-:xnack- 2022-11-23T01:34:41.6296010Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:41.6296398Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:41.6296767Z Default Rounding Mode: NEAR 2022-11-23T01:34:41.6297127Z Default Rounding Mode: NEAR 2022-11-23T01:34:41.6297476Z Fast f16: TRUE 2022-11-23T01:34:41.6297833Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:41.6298180Z Workgroup Max Size per Dimension: 2022-11-23T01:34:41.6298520Z x 1024(0x400) 2022-11-23T01:34:41.6298854Z y 1024(0x400) 2022-11-23T01:34:41.6299200Z z 1024(0x400) 2022-11-23T01:34:41.6299572Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:41.6299941Z Grid Max Size per Dimension: 2022-11-23T01:34:41.6300290Z x 4294967295(0xffffffff) 2022-11-23T01:34:41.6318920Z y 4294967295(0xffffffff) 2022-11-23T01:34:41.6319985Z z 4294967295(0xffffffff) 2022-11-23T01:34:41.6320854Z FBarrier Max Size: 32 2022-11-23T01:34:41.6321593Z *** Done *** 2022-11-23T01:34:41.6363914Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-11-23T01:34:41.6364329Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-11-23T01:34:41.6364664Z if [[ "x$ngpu" != "x2" && "x$ngpu" != "x4" ]]; then 2022-11-23T01:34:41.6364959Z  if [[ $ngpu -eq 0 ]]; then 2022-11-23T01:34:41.6365290Z  echo "Error: Failed to detect any GPUs on the runner" 2022-11-23T01:34:41.6365590Z  else 2022-11-23T01:34:41.6365933Z  echo "Error: Detected $ngpu GPUs on the runner, when only 2 or 4 were expected" 2022-11-23T01:34:41.6366370Z  fi 2022-11-23T01:34:41.6366800Z  echo "Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2022-11-23T01:34:41.6367216Z  exit 1 2022-11-23T01:34:41.6367432Z fi 2022-11-23T01:34:41.6398653Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:41.6398942Z env: 2022-11-23T01:34:41.6399179Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.6399500Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:41.6399776Z ##[endgroup] 2022-11-23T01:34:41.7484431Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:34:41.7485506Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:34:41.7486459Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:34:41.7548525Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:41.7549279Z env: 2022-11-23T01:34:41.7549901Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.7550708Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:41.7551404Z ##[endgroup] 2022-11-23T01:34:41.7818485Z ##[group]Run # Examine the runner name. If it ends with "-2", this is the second runner on the host. 2022-11-23T01:34:41.7819041Z # Examine the runner name. If it ends with "-2", this is the second runner on the host. 2022-11-23T01:34:41.7819457Z if [[ worker-rocm-amd-114 == *-2 ]]; then 2022-11-23T01:34:41.7819933Z  # select the last two GPUs on the host 2022-11-23T01:34:41.7820490Z  echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD130 --device=/dev/dri/renderD131 --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-11-23T01:34:41.7820972Z else 2022-11-23T01:34:41.7821273Z  # select the first two GPUs on the host 2022-11-23T01:34:41.7821811Z  echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-11-23T01:34:41.7822288Z fi 2022-11-23T01:34:41.7847622Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:41.7847937Z env: 2022-11-23T01:34:41.7848178Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.7848511Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:41.7848814Z ##[endgroup] 2022-11-23T01:34:41.7965055Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2022-11-23T01:34:41.7965641Z with: 2022-11-23T01:34:41.7966431Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:34:41.7967215Z env: 2022-11-23T01:34:41.7967629Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.7968169Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:41.7969030Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:41.7969768Z ##[endgroup] 2022-11-23T01:34:41.7994149Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:34:41.7994808Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:34:41.7995450Z # ignore output since only exit code is used for conditional 2022-11-23T01:34:41.7996311Z # only pull docker image if it's not available locally 2022-11-23T01:34:41.7997053Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2022-11-23T01:34:41.7997735Z  retry docker pull "${DOCKER_IMAGE}" 2022-11-23T01:34:41.7998188Z fi 2022-11-23T01:34:41.8038989Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:41.8039283Z env: 2022-11-23T01:34:41.8039540Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.8039873Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:41.8040393Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:41.8041087Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:34:41.8041563Z ##[endgroup] 2022-11-23T01:34:41.8812414Z ##[group]Run python3 -m pip install psutil==5.9.1 2022-11-23T01:34:41.8813375Z python3 -m pip install psutil==5.9.1 2022-11-23T01:34:41.8814215Z python3 -m pip install pynvml==11.4.1 2022-11-23T01:34:41.8815133Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2022-11-23T01:34:41.8816136Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2022-11-23T01:34:41.8882572Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:41.8883318Z env: 2022-11-23T01:34:41.8883927Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:41.8884731Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:41.8886009Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:41.8887090Z ##[endgroup] 2022-11-23T01:34:42.7499762Z Collecting psutil==5.9.1 2022-11-23T01:34:42.9870091Z Installing collected packages: psutil 2022-11-23T01:34:43.0896388Z Successfully installed psutil-5.9.4 2022-11-23T01:34:43.9452368Z Collecting pynvml==11.4.1 2022-11-23T01:34:43.9880854Z Using cached https://files.pythonhosted.org/packages/cc/0a/47be6726fd13f1f4371fa858b506228ed12bc418c07ffcaa6c0f7ceedac0/pynvml-11.4.1-py3-none-any.whl 2022-11-23T01:34:43.9910699Z Installing collected packages: pynvml 2022-11-23T01:34:44.0343258Z Successfully installed pynvml-11.4.1 2022-11-23T01:34:44.0826473Z Prepare all required actions 2022-11-23T01:34:44.0826821Z Getting action download info 2022-11-23T01:34:44.2966187Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea) 2022-11-23T01:34:45.3715553Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7) 2022-11-23T01:34:46.1737736Z ##[group]Run ./.github/actions/download-build-artifacts 2022-11-23T01:34:46.1738063Z with: 2022-11-23T01:34:46.1738343Z name: linux-focal-rocm5.2-py3.8 2022-11-23T01:34:46.1738601Z env: 2022-11-23T01:34:46.1738848Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:46.1739178Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:46.1739710Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:46.1740154Z ##[endgroup] 2022-11-23T01:34:46.1774801Z ##[group]Run seemethere/download-artifact-s3@v4 2022-11-23T01:34:46.1775124Z with: 2022-11-23T01:34:46.1775403Z name: linux-focal-rocm5.2-py3.8 2022-11-23T01:34:46.1775693Z s3-bucket: gha-artifacts 2022-11-23T01:34:46.1775980Z region: us-east-1 2022-11-23T01:34:46.1776230Z env: 2022-11-23T01:34:46.1776479Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:46.1776802Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:46.1777301Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:46.1777744Z ##[endgroup] 2022-11-23T01:34:46.9329368Z Found 1 objects with prefix pytorch/pytorch/3528394938/linux-focal-rocm5.2-py3.8/ 2022-11-23T01:34:46.9331110Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:35:36.9500834Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:35:36.9501681Z 2022-11-23T01:35:36.9547063Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-11-23T01:35:36.9570453Z Artifact download has finished successfully 2022-11-23T01:35:36.9710061Z ##[group]Run unzip -o artifacts.zip 2022-11-23T01:35:36.9710868Z unzip -o artifacts.zip 2022-11-23T01:35:36.9777275Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:35:36.9778023Z env: 2022-11-23T01:35:36.9778639Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:35:36.9779449Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:35:36.9780691Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:35:36.9781808Z ##[endgroup] 2022-11-23T01:35:36.9886949Z Archive: artifacts.zip 2022-11-23T01:35:36.9889005Z creating: dist/ 2022-11-23T01:35:38.1526763Z inflating: dist/torch-1.14.0a0+git1cfd385-cp38-cp38-linux_x86_64.whl 2022-11-23T01:35:38.1528656Z creating: build/custom_test_artifacts/ 2022-11-23T01:35:38.1530489Z creating: build/custom_test_artifacts/custom-op-build/ 2022-11-23T01:35:38.1532751Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2022-11-23T01:35:38.1535149Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:35:38.1537743Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2022-11-23T01:35:38.1540349Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:35:38.1542947Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:35:38.1546098Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:35:38.1547938Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:35:38.1549464Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:35:38.1550853Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:35:38.1552279Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:35:38.1553848Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:35:38.1555380Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:35:38.1557311Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:35:38.1559034Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:35:38.1560908Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:35:38.1563640Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:35:38.1565155Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:35:38.1566617Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2022-11-23T01:35:38.1568116Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2022-11-23T01:35:38.1569583Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2022-11-23T01:35:38.1571172Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2022-11-23T01:35:38.1572863Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2022-11-23T01:35:38.1574498Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2022-11-23T01:35:38.1576051Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2022-11-23T01:35:38.1577640Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2022-11-23T01:35:38.1579245Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2022-11-23T01:35:38.1580870Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2022-11-23T01:35:38.1582461Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2022-11-23T01:35:38.1584062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2022-11-23T01:35:38.1585691Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2022-11-23T01:35:38.1656096Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2022-11-23T01:35:38.1657876Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2022-11-23T01:35:38.1660720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2022-11-23T01:35:38.1663574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2022-11-23T01:35:38.1665297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2022-11-23T01:35:38.1666935Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2022-11-23T01:35:38.1668615Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2022-11-23T01:35:38.1670590Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2022-11-23T01:35:38.1672474Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2022-11-23T01:35:38.1674128Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2022-11-23T01:35:38.1675799Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2022-11-23T01:35:38.1677771Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2022-11-23T01:35:38.1742936Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2022-11-23T01:35:38.1744663Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:35:38.1745550Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:35:38.1746488Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2022-11-23T01:35:38.1747318Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2022-11-23T01:35:38.1748065Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2022-11-23T01:35:38.1748804Z inflating: build/custom_test_artifacts/custom-op-build/detect_rocm_version.cc 2022-11-23T01:35:38.1749510Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2022-11-23T01:35:38.1750187Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2022-11-23T01:35:38.1750880Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2022-11-23T01:35:38.1825478Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2022-11-23T01:35:38.1877673Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2022-11-23T01:35:38.1879036Z creating: build/custom_test_artifacts/jit-hook-build/ 2022-11-23T01:35:38.1880318Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2022-11-23T01:35:38.1881776Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:35:38.1883219Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2022-11-23T01:35:38.1884746Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:35:38.1886276Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:35:38.1887800Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:35:38.1889434Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:35:38.1891041Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:35:38.1892569Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:35:38.1894101Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:35:38.1895868Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:35:38.1897557Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:35:38.1899245Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:35:38.1900913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:35:38.1902638Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:35:38.1904321Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:35:38.1906157Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:35:38.1907596Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2022-11-23T01:35:38.1909224Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2022-11-23T01:35:38.1910709Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2022-11-23T01:35:38.1912326Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2022-11-23T01:35:38.1914030Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2022-11-23T01:35:38.1915664Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2022-11-23T01:35:38.1917374Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2022-11-23T01:35:38.1918432Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2022-11-23T01:35:38.1919449Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2022-11-23T01:35:38.1920494Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2022-11-23T01:35:38.1921513Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2022-11-23T01:35:38.1922529Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2022-11-23T01:35:38.1923603Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2022-11-23T01:35:38.1966862Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2022-11-23T01:35:38.1968067Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:35:38.1969175Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:35:38.1970160Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2022-11-23T01:35:38.1971082Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2022-11-23T01:35:38.1971989Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2022-11-23T01:35:38.1972904Z inflating: build/custom_test_artifacts/jit-hook-build/detect_rocm_version.cc 2022-11-23T01:35:38.1973782Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2022-11-23T01:35:38.1974595Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2022-11-23T01:35:38.1975438Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2022-11-23T01:35:38.2014462Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2022-11-23T01:35:38.2015358Z creating: build/custom_test_artifacts/custom-backend-build/ 2022-11-23T01:35:38.2016221Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2022-11-23T01:35:38.2017209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:35:38.2018184Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2022-11-23T01:35:38.2019186Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:35:38.2020191Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:35:38.2021216Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:35:38.2022316Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:35:38.2023391Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:35:38.2024633Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:35:38.2025644Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:35:38.2026875Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:35:38.2028007Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:35:38.2029096Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:35:38.2030211Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:35:38.2031367Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:35:38.2032503Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:35:38.2033507Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:35:38.2034474Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2022-11-23T01:35:38.2035469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2022-11-23T01:35:38.2036584Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2022-11-23T01:35:38.2037635Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2022-11-23T01:35:38.2038780Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2022-11-23T01:35:38.2039883Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2022-11-23T01:35:38.2040941Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2022-11-23T01:35:38.2042039Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2022-11-23T01:35:38.2043141Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2022-11-23T01:35:38.2044244Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2022-11-23T01:35:38.2045330Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2022-11-23T01:35:38.2046398Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2022-11-23T01:35:38.2047548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2022-11-23T01:35:38.2156014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2022-11-23T01:35:38.2157642Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2022-11-23T01:35:38.2159165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2022-11-23T01:35:38.2160730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2022-11-23T01:35:38.2162229Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2022-11-23T01:35:38.2163714Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2022-11-23T01:35:38.2165196Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2022-11-23T01:35:38.2166668Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2022-11-23T01:35:38.2168419Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2022-11-23T01:35:38.2169955Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2022-11-23T01:35:38.2171543Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2022-11-23T01:35:38.2176121Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2022-11-23T01:35:38.2224318Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2022-11-23T01:35:38.2225975Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:35:38.2227475Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:35:38.2228883Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2022-11-23T01:35:38.2230176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2022-11-23T01:35:38.2231452Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2022-11-23T01:35:38.2232724Z inflating: build/custom_test_artifacts/custom-backend-build/detect_rocm_version.cc 2022-11-23T01:35:38.2233957Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2022-11-23T01:35:38.2235127Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2022-11-23T01:35:38.2236499Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2022-11-23T01:35:38.2325672Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2022-11-23T01:35:38.2363234Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2022-11-23T01:35:38.2364247Z creating: build/lib/ 2022-11-23T01:35:38.2364951Z inflating: build/lib/libclog.a 2022-11-23T01:35:38.2423915Z inflating: build/lib/libgtest.a 2022-11-23T01:35:38.2430695Z inflating: build/lib/libpthreadpool.a 2022-11-23T01:35:38.2503038Z inflating: build/lib/libbenchmark.a 2022-11-23T01:35:38.2587483Z inflating: build/lib/libprotobuf-lite.a 2022-11-23T01:35:38.2592896Z inflating: build/lib/libittnotify.a 2022-11-23T01:35:38.2621227Z inflating: build/lib/libtensorpipe_uv.a 2022-11-23T01:35:38.2681999Z inflating: build/lib/libasmjit.a 2022-11-23T01:35:38.3112429Z inflating: build/lib/libprotobuf.a 2022-11-23T01:35:38.3201217Z inflating: build/lib/libgloo.a 2022-11-23T01:35:38.3228845Z inflating: build/lib/libfmt.a 2022-11-23T01:35:38.3229737Z inflating: build/lib/libcaffe2_nvrtc.so 2022-11-23T01:35:38.3230500Z inflating: build/lib/libfoxi_loader.a 2022-11-23T01:35:38.3288502Z inflating: build/lib/libc10.so 2022-11-23T01:35:38.3289436Z inflating: build/lib/libtorch_global_deps.so 2022-11-23T01:35:38.3296128Z inflating: build/lib/libcpuinfo.a 2022-11-23T01:35:38.3297941Z inflating: build/lib/libnnpack_reference_layers.a 2022-11-23T01:35:38.3305658Z inflating: build/lib/libcpuinfo_internals.a 2022-11-23T01:35:38.3324259Z inflating: build/lib/libgmock.a 2022-11-23T01:35:38.3325160Z inflating: build/lib/libgtest_main.a 2022-11-23T01:35:38.3325998Z inflating: build/lib/libbenchmark_main.a 2022-11-23T01:35:39.1180675Z inflating: build/lib/libdnnl.a 2022-11-23T01:35:39.1657014Z inflating: build/lib/libprotoc.a 2022-11-23T01:35:39.2194709Z inflating: build/lib/libtensorpipe.a 2022-11-23T01:35:39.2557976Z inflating: build/lib/libgloo_hip.a 2022-11-23T01:35:39.2593136Z inflating: build/lib/libc10_hip.so 2022-11-23T01:35:39.2594829Z inflating: build/lib/libgmock_main.a 2022-11-23T01:35:39.3758915Z inflating: build/lib/libfbgemm.a 2022-11-23T01:35:39.3770951Z inflating: build/lib/libqnnpack.a 2022-11-23T01:35:39.4695732Z inflating: build/lib/libdnnl_graph.a 2022-11-23T01:35:39.4967534Z inflating: build/lib/libkineto.a 2022-11-23T01:35:39.4986453Z inflating: build/lib/libpytorch_qnnpack.a 2022-11-23T01:35:39.5024017Z inflating: build/lib/libcaffe2_protos.a 2022-11-23T01:35:39.5134042Z inflating: build/lib/libXNNPACK.a 2022-11-23T01:35:39.5172889Z inflating: build/lib/libonnx_proto.a 2022-11-23T01:35:39.5719851Z inflating: build/lib/libonnx.a 2022-11-23T01:35:39.5737493Z inflating: build/lib/libnnpack.a 2022-11-23T01:35:41.7783034Z inflating: build/lib/libtorch_cpu.so 2022-11-23T01:35:42.8472661Z inflating: build/lib/libtorch_hip.so 2022-11-23T01:35:42.8474208Z inflating: build/lib/libtorch.so 2022-11-23T01:35:42.8494253Z inflating: build/lib/libjitbackend_test.so 2022-11-23T01:35:42.8543625Z inflating: build/lib/libtorchbind_test.so 2022-11-23T01:35:42.8568451Z inflating: build/lib/libbackend_with_compiler.so 2022-11-23T01:35:42.8570402Z inflating: build/lib/libshm.so 2022-11-23T01:35:43.0128474Z inflating: build/lib/libtorch_python.so 2022-11-23T01:35:43.0161779Z inflating: build/lib/libnnapi_backend.so 2022-11-23T01:35:43.0163369Z creating: build/bin/ 2022-11-23T01:35:43.0164662Z creating: build/bin/CMakeFiles/ 2022-11-23T01:35:43.0166386Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:35:43.0167461Z extracting: build/bin/CMakeFiles/progress.marks 2022-11-23T01:35:43.0209962Z inflating: build/bin/hip_generator_test 2022-11-23T01:35:43.0211568Z inflating: build/bin/Makefile 2022-11-23T01:35:43.0257428Z inflating: build/bin/variant_test 2022-11-23T01:35:43.0259010Z inflating: build/bin/cmake_install.cmake 2022-11-23T01:35:43.0306755Z inflating: build/bin/undefined_tensor_test 2022-11-23T01:35:43.0308396Z inflating: build/bin/CTestTestfile.cmake 2022-11-23T01:35:43.0354642Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-11-23T01:35:43.0402331Z inflating: build/bin/c10_DeviceGuard_test 2022-11-23T01:35:43.0450369Z inflating: build/bin/c10_Device_test 2022-11-23T01:35:43.0506096Z inflating: build/bin/c10_DispatchKeySet_test 2022-11-23T01:35:43.0551956Z inflating: build/bin/c10_StreamGuard_test 2022-11-23T01:35:43.0599050Z inflating: build/bin/c10_SymInt_test 2022-11-23T01:35:43.0650470Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-11-23T01:35:43.0703050Z inflating: build/bin/c10_InlineStreamGuard_test 2022-11-23T01:35:43.0756302Z inflating: build/bin/c10_SizesAndStrides_test 2022-11-23T01:35:43.0802056Z inflating: build/bin/c10_Array_test 2022-11-23T01:35:43.0852801Z inflating: build/bin/c10_Bitset_test 2022-11-23T01:35:43.0898817Z inflating: build/bin/c10_ConstexprCrc_test 2022-11-23T01:35:43.0947294Z inflating: build/bin/c10_C++17_test 2022-11-23T01:35:43.0994386Z inflating: build/bin/c10_DeadlockDetection_test 2022-11-23T01:35:43.1041213Z inflating: build/bin/c10_Half_test 2022-11-23T01:35:43.1094281Z inflating: build/bin/c10_LeftRight_test 2022-11-23T01:35:43.1154885Z inflating: build/bin/c10_Metaprogramming_test 2022-11-23T01:35:43.1202205Z inflating: build/bin/c10_Synchronized_test 2022-11-23T01:35:43.1344249Z inflating: build/bin/c10_SmallVectorTest 2022-11-23T01:35:43.1396996Z inflating: build/bin/c10_ThreadLocal_test 2022-11-23T01:35:43.1445463Z inflating: build/bin/c10_TypeIndex_test 2022-11-23T01:35:43.1491666Z inflating: build/bin/c10_TypeTraits_test 2022-11-23T01:35:43.1539563Z inflating: build/bin/c10_TypeList_test 2022-11-23T01:35:43.1589496Z inflating: build/bin/c10_accumulate_test 2022-11-23T01:35:43.1641966Z inflating: build/bin/c10_bfloat16_test 2022-11-23T01:35:43.1693535Z inflating: build/bin/c10_complex_math_test 2022-11-23T01:35:43.1741030Z inflating: build/bin/c10_flags_test 2022-11-23T01:35:43.1790831Z inflating: build/bin/c10_exception_test 2022-11-23T01:35:43.1842302Z inflating: build/bin/c10_complex_test 2022-11-23T01:35:43.1946415Z inflating: build/bin/c10_either_test 2022-11-23T01:35:43.1994205Z inflating: build/bin/c10_irange_test 2022-11-23T01:35:43.2047548Z inflating: build/bin/c10_logging_test 2022-11-23T01:35:43.2203437Z inflating: build/bin/c10_intrusive_ptr_test 2022-11-23T01:35:43.2273091Z inflating: build/bin/c10_optional_test 2022-11-23T01:35:43.2324716Z inflating: build/bin/c10_registry_test 2022-11-23T01:35:43.2383939Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-11-23T01:35:43.2432602Z inflating: build/bin/c10_tempfile_test 2022-11-23T01:35:43.2488834Z inflating: build/bin/c10_string_view_test 2022-11-23T01:35:43.2541878Z inflating: build/bin/c10_typeid_test 2022-11-23T01:35:43.2588562Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-11-23T01:35:43.3014880Z inflating: build/bin/protoc-3.13.0.0 2022-11-23T01:35:43.3440727Z inflating: build/bin/protoc 2022-11-23T01:35:43.3487212Z inflating: build/bin/c10_hip_HIPTest 2022-11-23T01:35:43.3760512Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-11-23T01:35:43.4052182Z inflating: build/bin/vec_test_all_types_AVX512 2022-11-23T01:35:43.4358563Z inflating: build/bin/vec_test_all_types_AVX2 2022-11-23T01:35:43.4408660Z inflating: build/bin/HashStoreTest 2022-11-23T01:35:43.4458609Z inflating: build/bin/FileStoreTest 2022-11-23T01:35:43.4515998Z inflating: build/bin/TCPStoreTest 2022-11-23T01:35:43.4517968Z inflating: build/bin/example_allreduce 2022-11-23T01:35:43.4580028Z inflating: build/bin/ProcessGroupGlooTest 2022-11-23T01:35:43.4629277Z inflating: build/bin/Dimname_test 2022-11-23T01:35:43.4698478Z inflating: build/bin/Dict_test 2022-11-23T01:35:43.4752725Z inflating: build/bin/NamedTensor_test 2022-11-23T01:35:43.4815581Z inflating: build/bin/MaybeOwned_test 2022-11-23T01:35:43.4872934Z inflating: build/bin/static_runtime_bench 2022-11-23T01:35:43.4928082Z inflating: build/bin/apply_utils_test 2022-11-23T01:35:43.4987030Z inflating: build/bin/basic 2022-11-23T01:35:43.5042372Z inflating: build/bin/atest 2022-11-23T01:35:43.5094019Z inflating: build/bin/broadcast_test 2022-11-23T01:35:43.5148864Z inflating: build/bin/cpu_generator_test 2022-11-23T01:35:43.5199568Z inflating: build/bin/cpu_profiling_allocator_test 2022-11-23T01:35:43.5442055Z inflating: build/bin/static_runtime_test 2022-11-23T01:35:43.5489419Z inflating: build/bin/dispatch_key_set_test 2022-11-23T01:35:43.5573391Z inflating: build/bin/cpu_rng_test 2022-11-23T01:35:43.5620984Z inflating: build/bin/dlconvertor_test 2022-11-23T01:35:43.5675759Z inflating: build/bin/extension_backend_test 2022-11-23T01:35:43.5727910Z inflating: build/bin/half_test 2022-11-23T01:35:43.5774985Z inflating: build/bin/lazy_tensor_test 2022-11-23T01:35:43.5825715Z inflating: build/bin/math_kernel_test 2022-11-23T01:35:43.5914769Z inflating: build/bin/ivalue_test 2022-11-23T01:35:43.5966526Z inflating: build/bin/memory_format_test 2022-11-23T01:35:43.6016408Z inflating: build/bin/memory_overlapping_test 2022-11-23T01:35:43.6064195Z inflating: build/bin/operator_name_test 2022-11-23T01:35:43.6114510Z inflating: build/bin/mobile_memory_cleanup 2022-11-23T01:35:43.6168364Z inflating: build/bin/native_test 2022-11-23T01:35:43.6216955Z inflating: build/bin/operators_test 2022-11-23T01:35:43.6265847Z inflating: build/bin/packedtensoraccessor_test 2022-11-23T01:35:43.6319211Z inflating: build/bin/quantized_test 2022-11-23T01:35:43.6366249Z inflating: build/bin/reduce_ops_test 2022-11-23T01:35:43.6414494Z inflating: build/bin/reportMemoryUsage_test 2022-11-23T01:35:43.6476620Z inflating: build/bin/pow_test 2022-11-23T01:35:43.6531160Z inflating: build/bin/scalar_tensor_test 2022-11-23T01:35:43.6585149Z inflating: build/bin/scalar_test 2022-11-23T01:35:43.6634658Z inflating: build/bin/stride_properties_test 2022-11-23T01:35:43.6637122Z inflating: build/bin/thread_init_test 2022-11-23T01:35:43.6691201Z inflating: build/bin/type_ptr_test 2022-11-23T01:35:43.6764725Z inflating: build/bin/tensor_iterator_test 2022-11-23T01:35:43.6816077Z inflating: build/bin/test_parallel 2022-11-23T01:35:43.6873761Z inflating: build/bin/type_test 2022-11-23T01:35:43.6875167Z inflating: build/bin/verify_api_visibility 2022-11-23T01:35:43.6941388Z inflating: build/bin/vmap_test 2022-11-23T01:35:43.6990107Z inflating: build/bin/weakref_test 2022-11-23T01:35:43.7038763Z inflating: build/bin/wrapdim_test 2022-11-23T01:35:43.7085362Z inflating: build/bin/xla_tensor_test 2022-11-23T01:35:43.7186082Z inflating: build/bin/List_test 2022-11-23T01:35:43.7242868Z inflating: build/bin/IListRef_test 2022-11-23T01:35:43.7356778Z inflating: build/bin/kernel_function_legacy_test 2022-11-23T01:35:43.7419053Z inflating: build/bin/KernelFunction_test 2022-11-23T01:35:43.7508770Z inflating: build/bin/kernel_function_test 2022-11-23T01:35:43.7627650Z inflating: build/bin/kernel_lambda_legacy_test 2022-11-23T01:35:43.7724152Z inflating: build/bin/kernel_lambda_test 2022-11-23T01:35:43.7780681Z inflating: build/bin/kernel_stackbased_test 2022-11-23T01:35:43.7870474Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-11-23T01:35:43.7918627Z inflating: build/bin/CppSignature_test 2022-11-23T01:35:43.7964118Z inflating: build/bin/op_allowlist_test 2022-11-23T01:35:43.8017264Z inflating: build/bin/backend_fallback_test 2022-11-23T01:35:43.8063188Z inflating: build/bin/hip_complex_math_test 2022-11-23T01:35:43.8113354Z inflating: build/bin/inline_container_test 2022-11-23T01:35:43.8392430Z inflating: build/bin/op_registration_test 2022-11-23T01:35:43.8438569Z inflating: build/bin/hip_complex_test 2022-11-23T01:35:43.8488354Z inflating: build/bin/hip_apply_test 2022-11-23T01:35:43.8534409Z inflating: build/bin/hip_distributions_test 2022-11-23T01:35:43.8580235Z inflating: build/bin/hip_half_test 2022-11-23T01:35:43.8626285Z inflating: build/bin/hip_integer_divider_test 2022-11-23T01:35:43.8672091Z inflating: build/bin/hip_optional_test 2022-11-23T01:35:43.8718198Z inflating: build/bin/hip_packedtensoraccessor_test 2022-11-23T01:35:43.8766066Z inflating: build/bin/hip_dlconvertor_test 2022-11-23T01:35:43.8811897Z inflating: build/bin/hip_vectorized_test 2022-11-23T01:35:43.8827213Z inflating: build/bin/tutorial_tensorexpr 2022-11-23T01:35:43.8879479Z inflating: build/bin/test_dist_autograd 2022-11-23T01:35:43.8944862Z inflating: build/bin/test_cpp_rpc 2022-11-23T01:35:43.9700016Z inflating: build/bin/test_tensorexpr 2022-11-23T01:35:43.9701803Z inflating: build/bin/parallel_benchmark 2022-11-23T01:35:43.9766167Z inflating: build/bin/test_mobile_nnc 2022-11-23T01:35:43.9773752Z inflating: build/bin/aot_model_compiler_test 2022-11-23T01:35:44.0112928Z inflating: build/bin/test_lazy 2022-11-23T01:35:44.1201822Z inflating: build/bin/test_api 2022-11-23T01:35:44.1204914Z inflating: build/bin/torch_shm_manager 2022-11-23T01:35:44.1736125Z inflating: build/bin/test_jit 2022-11-23T01:35:44.1737483Z inflating: .pytorch-test-times.json 2022-11-23T01:35:44.1788195Z ##[group]Run df -H 2022-11-23T01:35:44.1788834Z df -H 2022-11-23T01:35:44.1852210Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:35:44.1852952Z env: 2022-11-23T01:35:44.1853550Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:35:44.1854335Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:35:44.1855614Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:35:44.1856691Z ##[endgroup] 2022-11-23T01:35:44.1939858Z Filesystem Size Used Avail Use% Mounted on 2022-11-23T01:35:44.1940817Z udev 271G 0 271G 0% /dev 2022-11-23T01:35:44.1942057Z tmpfs 55G 2.3M 55G 1% /run 2022-11-23T01:35:44.1943469Z /dev/mapper/ubuntu--server--x8664--vg-root 944G 238G 658G 27% / 2022-11-23T01:35:44.1944442Z tmpfs 271G 17k 271G 1% /dev/shm 2022-11-23T01:35:44.1945450Z tmpfs 5.3M 0 5.3M 0% /run/lock 2022-11-23T01:35:44.1946331Z tmpfs 271G 0 271G 0% /sys/fs/cgroup 2022-11-23T01:35:44.1947191Z /dev/sda1 755M 596M 121M 84% /boot 2022-11-23T01:35:44.1948048Z tmpfs 55G 13k 55G 1% /run/user/1121 2022-11-23T01:35:44.1948874Z tmpfs 55G 0 55G 0% /run/user/1000 2022-11-23T01:35:44.1996673Z ##[group]Run .github/scripts/parse_ref.py 2022-11-23T01:35:44.1997509Z .github/scripts/parse_ref.py 2022-11-23T01:35:44.2045913Z shell: /bin/bash -e {0} 2022-11-23T01:35:44.2046169Z env: 2022-11-23T01:35:44.2046412Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:35:44.2046718Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:35:44.2047224Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:35:44.2047656Z ##[endgroup] 2022-11-23T01:35:44.2506134Z ##[group]Run set -x 2022-11-23T01:35:44.2506985Z set -x 2022-11-23T01:35:44.2507557Z  2022-11-23T01:35:44.2508255Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-11-23T01:35:44.2509155Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-11-23T01:35:44.2510094Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-11-23T01:35:44.2510946Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-11-23T01:35:44.2511643Z else 2022-11-23T01:35:44.2512356Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:35:44.2513049Z fi 2022-11-23T01:35:44.2513580Z  2022-11-23T01:35:44.2514414Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2022-11-23T01:35:44.2515244Z  2022-11-23T01:35:44.2515999Z # sanitize the input commit message and PR body here: 2022-11-23T01:35:44.2516923Z # 2022-11-23T01:35:44.2517856Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2022-11-23T01:35:44.2519191Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2022-11-23T01:35:44.2520323Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2022-11-23T01:35:44.2521167Z PR_BODY="${PR_BODY//[$'\n\r']}" 2022-11-23T01:35:44.2521819Z  2022-11-23T01:35:44.2522731Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2022-11-23T01:35:44.2523706Z # wreak havoc internally 2022-11-23T01:35:44.2524521Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2022-11-23T01:35:44.2525389Z export PR_BODY="${PR_BODY//[\'\"]}" 2022-11-23T01:35:44.2526042Z  2022-11-23T01:35:44.2562434Z # detached container should get cleaned up by teardown_ec2_linux 2022-11-23T01:35:44.2562950Z # TODO: Stop building test binaries as part of the build phase 2022-11-23T01:35:44.2563358Z # Used for GPU_FLAG since that doesn't play nice 2022-11-23T01:35:44.2563710Z # shellcheck disable=SC2086,SC2090 2022-11-23T01:35:44.2564032Z container_name=$(docker run \ 2022-11-23T01:35:44.2564328Z  ${GPU_FLAG:-} \ 2022-11-23T01:35:44.2564613Z  -e BUILD_ENVIRONMENT \ 2022-11-23T01:35:44.2564882Z  -e PR_NUMBER \ 2022-11-23T01:35:44.2565158Z  -e GITHUB_ACTIONS \ 2022-11-23T01:35:44.2565428Z  -e BRANCH \ 2022-11-23T01:35:44.2565685Z  -e SHA1 \ 2022-11-23T01:35:44.2565958Z  -e AWS_DEFAULT_REGION \ 2022-11-23T01:35:44.2566575Z  -e IN_WHEEL_TEST \ 2022-11-23T01:35:44.2566831Z  -e SHARD_NUMBER \ 2022-11-23T01:35:44.2567105Z  -e TEST_CONFIG \ 2022-11-23T01:35:44.2567385Z  -e NUM_TEST_SHARDS \ 2022-11-23T01:35:44.2567653Z  -e PR_BODY \ 2022-11-23T01:35:44.2567927Z  -e COMMIT_MESSAGES \ 2022-11-23T01:35:44.2568236Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-11-23T01:35:44.2568549Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2022-11-23T01:35:44.2568881Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-11-23T01:35:44.2569186Z  -e SCCACHE_BUCKET \ 2022-11-23T01:35:44.2569498Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-11-23T01:35:44.2569838Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2022-11-23T01:35:44.2570182Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2022-11-23T01:35:44.2570536Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-11-23T01:35:44.2570877Z  --ulimit stack=10485760:83886080 \ 2022-11-23T01:35:44.2571209Z  --security-opt seccomp=unconfined \ 2022-11-23T01:35:44.2571528Z  --cap-add=SYS_PTRACE \ 2022-11-23T01:35:44.2571814Z  --shm-size="8g" \ 2022-11-23T01:35:44.2572067Z  --tty \ 2022-11-23T01:35:44.2572299Z  --detach \ 2022-11-23T01:35:44.2572676Z  --name="${container_name}" \ 2022-11-23T01:35:44.2572961Z  --user jenkins \ 2022-11-23T01:35:44.2573295Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-11-23T01:35:44.2573660Z  -w /var/lib/jenkins/workspace \ 2022-11-23T01:35:44.2573955Z  "${DOCKER_IMAGE}" 2022-11-23T01:35:44.2574187Z ) 2022-11-23T01:35:44.2574465Z # save container name for later step 2022-11-23T01:35:44.2574827Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2022-11-23T01:35:44.2575329Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2022-11-23T01:35:44.2575935Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2022-11-23T01:35:44.2601278Z shell: /bin/bash -e {0} 2022-11-23T01:35:44.2601519Z env: 2022-11-23T01:35:44.2601792Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:35:44.2602115Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:35:44.2602621Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:35:44.2603126Z BUILD_ENVIRONMENT: linux-focal-rocm5.2-py3.8 2022-11-23T01:35:44.2603440Z PR_NUMBER: 2022-11-23T01:35:44.2603666Z BRANCH: master 2022-11-23T01:35:44.2603957Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:35:44.2604270Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T01:35:44.2604566Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T01:35:44.2604865Z TEST_CONFIG: distributed 2022-11-23T01:35:44.2605132Z SHARD_NUMBER: 1 2022-11-23T01:35:44.2605364Z NUM_TEST_SHARDS: 2 2022-11-23T01:35:44.2605611Z PR_BODY: 2022-11-23T01:35:44.2605935Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-11-23T01:35:44.2606509Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:35:44.2607120Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:35:44.2607526Z PYTORCH_JIT_ENABLE_NVFUSER: 1 2022-11-23T01:35:44.2607817Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2022-11-23T01:35:44.2608131Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 1 2022-11-23T01:35:44.2608417Z ##[endgroup] 2022-11-23T01:35:44.2663390Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-11-23T01:35:44.2664600Z + [[ linux-focal-rocm5.2-py3.8 == *onnx* ]] 2022-11-23T01:35:44.2665563Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:35:44.2671819Z ++ git cherry -v origin/master 2022-11-23T01:35:44.2710935Z + COMMIT_MESSAGES= 2022-11-23T01:35:44.2711572Z + COMMIT_MESSAGES= 2022-11-23T01:35:44.2712120Z + PR_BODY= 2022-11-23T01:35:44.2712670Z + export COMMIT_MESSAGES= 2022-11-23T01:35:44.2713275Z + COMMIT_MESSAGES= 2022-11-23T01:35:44.2713824Z + export PR_BODY= 2022-11-23T01:35:44.2714348Z + PR_BODY= 2022-11-23T01:35:44.2735108Z +++ nproc --ignore=2 2022-11-23T01:35:44.2765468Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e MAX_JOBS=54 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528394938 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:35:47.4302153Z + container_name=de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T01:35:47.4302736Z + echo CONTAINER_NAME=de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T01:35:47.4304451Z + docker exec -t de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .jenkins/pytorch/test.sh' 2022-11-23T01:35:53.5178505Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp38-cp38-linux_x86_64.whl 2022-11-23T01:35:54.0029809Z Requirement already satisfied: sympy in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (1.11.1) 2022-11-23T01:35:54.0031853Z Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3) 2022-11-23T01:35:54.0033924Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (4.4.0) 2022-11-23T01:35:54.0176735Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1) 2022-11-23T01:35:54.6356967Z Installing collected packages: torch 2022-11-23T01:36:00.6991960Z Successfully installed torch-1.14.0a0+git1cfd385 2022-11-23T01:36:00.7615468Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-11-23T01:36:00.7921370Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.8/site-packages/torch 2022-11-23T01:36:00.7922896Z + TORCH_BIN_DIR=/opt/conda/lib/python3.8/site-packages/torch/bin 2022-11-23T01:36:00.7924282Z + TORCH_LIB_DIR=/opt/conda/lib/python3.8/site-packages/torch/lib 2022-11-23T01:36:00.7925570Z + TORCH_TEST_DIR=/opt/conda/lib/python3.8/site-packages/torch/test 2022-11-23T01:36:00.7926408Z + BUILD_DIR=build 2022-11-23T01:36:00.7927083Z + BUILD_RENAMED_DIR=build_renamed 2022-11-23T01:36:00.7927763Z + BUILD_BIN_DIR=build/bin 2022-11-23T01:36:00.7928412Z + export VALGRIND=ON 2022-11-23T01:36:00.7929016Z + VALGRIND=ON 2022-11-23T01:36:00.7929959Z + [[ linux-focal-rocm5.2-py3.8 == *clang9* ]] 2022-11-23T01:36:00.7931021Z + [[ linux-focal-rocm5.2-py3.8 != *bazel* ]] 2022-11-23T01:36:00.7932839Z ++ realpath build/custom_test_artifacts 2022-11-23T01:36:00.7954050Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2022-11-23T01:36:00.7963154Z ++ dirname .jenkins/pytorch/test.sh 2022-11-23T01:36:00.7984300Z + source .jenkins/pytorch/common.sh 2022-11-23T01:36:00.7989380Z +++ dirname .jenkins/pytorch/common.sh 2022-11-23T01:36:00.7999744Z ++ source .jenkins/pytorch/common_utils.sh 2022-11-23T01:36:00.8002102Z +++ declare -f -t trap_add 2022-11-23T01:36:00.8021624Z ++ set -ex 2022-11-23T01:36:00.8022750Z ++ [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:36:00.8023579Z ++ unset HIP_PLATFORM 2022-11-23T01:36:00.8024303Z ++ export PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:36:00.8025019Z ++ PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:36:00.8025718Z ++ export HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:36:00.8026389Z ++ HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:36:00.8027089Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:36:00.8027821Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:36:00.8028502Z ++ BUILD_TEST_LIBTORCH=0 2022-11-23T01:36:00.8029366Z + echo 'Environment variables' 2022-11-23T01:36:00.8030036Z Environment variables 2022-11-23T01:36:00.8030611Z + env 2022-11-23T01:36:00.8045816Z INSTALLED_DB=yes 2022-11-23T01:36:00.8047379Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:36:00.8048888Z BUILD_ENVIRONMENT=linux-focal-rocm5.2-py3.8 2022-11-23T01:36:00.8049915Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2022-11-23T01:36:00.8050800Z HOSTNAME=de3c05a7b135 2022-11-23T01:36:00.8052697Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_5c1fa7c4-cbbc-4fd0-9566-c72e5d671aac 2022-11-23T01:36:00.8053882Z GITHUB_ACTION=__self 2022-11-23T01:36:00.8054595Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2022-11-23T01:36:00.8055298Z GITHUB_RUN_NUMBER=3445 2022-11-23T01:36:00.8056280Z TEST_CONFIG=distributed 2022-11-23T01:36:00.8057039Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2022-11-23T01:36:00.8057760Z GITHUB_REF_TYPE=branch 2022-11-23T01:36:00.8059470Z *** 2022-11-23T01:36:00.8060175Z GITHUB_ACTIONS=true 2022-11-23T01:36:00.8060912Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:36:00.8061757Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:36:00.8062535Z GITHUB_REF=refs/heads/master 2022-11-23T01:36:00.8063176Z SHARD_NUMBER=1 2022-11-23T01:36:00.8063808Z GITHUB_REF_PROTECTED=true 2022-11-23T01:36:00.8064440Z HOME=/var/lib/jenkins 2022-11-23T01:36:00.8065191Z GITHUB_API_URL=https://api.github.com 2022-11-23T01:36:00.8066018Z PYTORCH_TEST_RERUN_DISABLED_TESTS=1 2022-11-23T01:36:00.8066812Z LANG=C.UTF-8 2022-11-23T01:36:00.8067439Z PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:36:00.8068053Z NUM_TEST_SHARDS=2 2022-11-23T01:36:00.8069551Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_5c1fa7c4-cbbc-4fd0-9566-c72e5d671aac 2022-11-23T01:36:00.8070648Z MAGMA_HOME=/opt/rocm/magma 2022-11-23T01:36:00.8071337Z PYTORCH_RETRY_TEST_CASES=1 2022-11-23T01:36:00.8072819Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_5c1fa7c4-cbbc-4fd0-9566-c72e5d671aac 2022-11-23T01:36:00.8073848Z HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:36:00.8075085Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2022-11-23T01:36:00.8076028Z GITHUB_EVENT_NAME=schedule 2022-11-23T01:36:00.8076958Z GITHUB_RUN_ID=3528394938 2022-11-23T01:36:00.8077944Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_5c1fa7c4-cbbc-4fd0-9566-c72e5d671aac 2022-11-23T01:36:00.8078534Z GITHUB_ACTOR=pytorchmergebot 2022-11-23T01:36:00.8078868Z PR_NUMBER= 2022-11-23T01:36:00.8079162Z GITHUB_RUN_ATTEMPT=1 2022-11-23T01:36:00.8079473Z VALGRIND=ON 2022-11-23T01:36:00.8079873Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2022-11-23T01:36:00.8080263Z TERM=xterm 2022-11-23T01:36:00.8080570Z INSTALLED_VISION=yes 2022-11-23T01:36:00.8080881Z BRANCH=master 2022-11-23T01:36:00.8081530Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2022-11-23T01:36:00.8082078Z GITHUB_SERVER_URL=https://github.com 2022-11-23T01:36:00.8082466Z PYTORCH_ROCM_ARCH=gfx906 2022-11-23T01:36:00.8082784Z SHLVL=1 2022-11-23T01:36:00.8083055Z MAX_JOBS=54 2022-11-23T01:36:00.8083341Z COMMIT_MESSAGES= 2022-11-23T01:36:00.8083662Z GITHUB_REF_NAME=master 2022-11-23T01:36:00.8084282Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:36:00.8084752Z GITHUB_JOB=test 2022-11-23T01:36:00.8085236Z GITHUB_REPOSITORY=pytorch/pytorch 2022-11-23T01:36:00.8085616Z LC_ALL=C.UTF-8 2022-11-23T01:36:00.8085921Z GITHUB_RETENTION_DAYS=90 2022-11-23T01:36:00.8086270Z GITHUB_ACTION_REPOSITORY= 2022-11-23T01:36:00.8086924Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:36:00.8087500Z GITHUB_BASE_REF= 2022-11-23T01:36:00.8087791Z CI=true 2022-11-23T01:36:00.8088097Z HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:36:00.8088451Z GITHUB_REPOSITORY_OWNER=pytorch 2022-11-23T01:36:00.8088810Z INSTALLED_PROTOBUF=yes 2022-11-23T01:36:00.8089132Z GITHUB_HEAD_REF= 2022-11-23T01:36:00.8089444Z GITHUB_ACTION_REF= 2022-11-23T01:36:00.8089966Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2022-11-23T01:36:00.8090383Z GITHUB_WORKFLOW=periodic 2022-11-23T01:36:00.8090744Z DEBIAN_FRONTEND=noninteractive 2022-11-23T01:36:00.8091531Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_5c1fa7c4-cbbc-4fd0-9566-c72e5d671aac 2022-11-23T01:36:00.8092084Z OLDPWD=/var/lib/jenkins 2022-11-23T01:36:00.8092395Z PR_BODY= 2022-11-23T01:36:00.8092679Z _=/usr/bin/env 2022-11-23T01:36:00.8093047Z + echo 'Testing pytorch' 2022-11-23T01:36:00.8093374Z Testing pytorch 2022-11-23T01:36:00.8093725Z + export LANG=C.UTF-8 2022-11-23T01:36:00.8094145Z + LANG=C.UTF-8 2022-11-23T01:36:00.8094441Z + PR_NUMBER= 2022-11-23T01:36:00.8094762Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-11-23T01:36:00.8095160Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2022-11-23T01:36:00.8095684Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:36:00.8096092Z + export HIP_VISIBLE_DEVICES=0,1 2022-11-23T01:36:00.8096451Z + HIP_VISIBLE_DEVICES=0,1 2022-11-23T01:36:00.8096797Z + [[ distributed == \s\l\o\w ]] 2022-11-23T01:36:00.8097312Z + [[ linux-focal-rocm5.2-py3.8 == *slow-gradcheck* ]] 2022-11-23T01:36:00.8097865Z + [[ linux-focal-rocm5.2-py3.8 == *cuda* ]] 2022-11-23T01:36:00.8098387Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:36:00.8098818Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-11-23T01:36:00.8099234Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-11-23T01:36:00.8099604Z + [[ distributed == *crossref* ]] 2022-11-23T01:36:00.8099961Z + [[ distributed == *dynamo* ]] 2022-11-23T01:36:00.8100322Z + [[ distributed == *inductor* ]] 2022-11-23T01:36:00.8100813Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:36:00.8101168Z + rocminfo 2022-11-23T01:36:00.8327706Z ROCk module is loaded 2022-11-23T01:36:00.8935253Z ===================== 2022-11-23T01:36:00.8936160Z HSA System Attributes 2022-11-23T01:36:00.8936836Z ===================== 2022-11-23T01:36:00.8937584Z Runtime Version: 1.1 2022-11-23T01:36:00.8938445Z System Timestamp Freq.: 1000.000000MHz 2022-11-23T01:36:00.8939481Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2022-11-23T01:36:00.8940522Z Machine Model: LARGE 2022-11-23T01:36:00.8941225Z System Endianness: LITTLE 2022-11-23T01:36:00.8941728Z 2022-11-23T01:36:00.8941914Z ========== 2022-11-23T01:36:00.8942478Z HSA Agents 2022-11-23T01:36:00.8943022Z ========== 2022-11-23T01:36:00.8943478Z ******* 2022-11-23T01:36:00.8943946Z Agent 1 2022-11-23T01:36:00.8944383Z ******* 2022-11-23T01:36:00.8944966Z Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:36:00.8945941Z Uuid: CPU-XX 2022-11-23T01:36:00.8946630Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:36:00.8947330Z Vendor Name: CPU 2022-11-23T01:36:00.8947975Z Feature: None specified 2022-11-23T01:36:00.8948635Z Profile: FULL_PROFILE 2022-11-23T01:36:00.8949534Z Float Round Mode: NEAR 2022-11-23T01:36:00.8950160Z Max Queue Number: 0(0x0) 2022-11-23T01:36:00.8950773Z Queue Min Size: 0(0x0) 2022-11-23T01:36:00.8951396Z Queue Max Size: 0(0x0) 2022-11-23T01:36:00.8952009Z Queue Type: MULTI 2022-11-23T01:36:00.8952594Z Node: 0 2022-11-23T01:36:00.8953157Z Device Type: CPU 2022-11-23T01:36:00.8953697Z Cache Info: 2022-11-23T01:36:00.8954266Z L1: 32768(0x8000) KB 2022-11-23T01:36:00.8954873Z Chip ID: 0(0x0) 2022-11-23T01:36:00.8955466Z Cacheline Size: 64(0x40) 2022-11-23T01:36:00.8956302Z Max Clock Freq. (MHz): 3700 2022-11-23T01:36:00.8956877Z BDFID: 0 2022-11-23T01:36:00.8957470Z Internal Node ID: 0 2022-11-23T01:36:00.8958067Z Compute Unit: 28 2022-11-23T01:36:00.8958791Z SIMDs per CU: 0 2022-11-23T01:36:00.8959396Z Shader Engines: 0 2022-11-23T01:36:00.8960022Z Shader Arrs. per Eng.: 0 2022-11-23T01:36:00.8960661Z WatchPts on Addr. Ranges:1 2022-11-23T01:36:00.8961205Z Features: None 2022-11-23T01:36:00.8961709Z Pool Info: 2022-11-23T01:36:00.8962206Z Pool 1 2022-11-23T01:36:00.8962828Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:36:00.8963681Z Size: 263559840(0xfb59aa0) KB 2022-11-23T01:36:00.8964328Z Allocatable: TRUE 2022-11-23T01:36:00.8964980Z Alloc Granule: 4KB 2022-11-23T01:36:00.8965614Z Alloc Alignment: 4KB 2022-11-23T01:36:00.8966265Z Accessible by all: TRUE 2022-11-23T01:36:00.8966804Z Pool 2 2022-11-23T01:36:00.8967418Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:36:00.8968098Z Size: 263559840(0xfb59aa0) KB 2022-11-23T01:36:00.8968739Z Allocatable: TRUE 2022-11-23T01:36:00.8969378Z Alloc Granule: 4KB 2022-11-23T01:36:00.8970019Z Alloc Alignment: 4KB 2022-11-23T01:36:00.8970643Z Accessible by all: TRUE 2022-11-23T01:36:00.8971199Z Pool 3 2022-11-23T01:36:00.8971806Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:36:00.8972577Z Size: 263559840(0xfb59aa0) KB 2022-11-23T01:36:00.8973226Z Allocatable: TRUE 2022-11-23T01:36:00.8973860Z Alloc Granule: 4KB 2022-11-23T01:36:00.8974488Z Alloc Alignment: 4KB 2022-11-23T01:36:00.8975104Z Accessible by all: TRUE 2022-11-23T01:36:00.8975660Z ISA Info: 2022-11-23T01:36:00.8976140Z ******* 2022-11-23T01:36:00.8976603Z Agent 2 2022-11-23T01:36:00.8977066Z ******* 2022-11-23T01:36:00.8977636Z Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:36:00.8978626Z Uuid: CPU-XX 2022-11-23T01:36:00.8979300Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:36:00.8979978Z Vendor Name: CPU 2022-11-23T01:36:00.8980618Z Feature: None specified 2022-11-23T01:36:00.8981270Z Profile: FULL_PROFILE 2022-11-23T01:36:00.8981909Z Float Round Mode: NEAR 2022-11-23T01:36:00.8982530Z Max Queue Number: 0(0x0) 2022-11-23T01:36:00.8983156Z Queue Min Size: 0(0x0) 2022-11-23T01:36:00.8983769Z Queue Max Size: 0(0x0) 2022-11-23T01:36:00.8984384Z Queue Type: MULTI 2022-11-23T01:36:00.8984976Z Node: 1 2022-11-23T01:36:00.8985568Z Device Type: CPU 2022-11-23T01:36:00.8986089Z Cache Info: 2022-11-23T01:36:00.8986649Z L1: 32768(0x8000) KB 2022-11-23T01:36:00.8987250Z Chip ID: 0(0x0) 2022-11-23T01:36:00.8987946Z Cacheline Size: 64(0x40) 2022-11-23T01:36:00.8988565Z Max Clock Freq. (MHz): 3700 2022-11-23T01:36:00.8989159Z BDFID: 0 2022-11-23T01:36:00.8989727Z Internal Node ID: 1 2022-11-23T01:36:00.8990325Z Compute Unit: 28 2022-11-23T01:36:00.8990915Z SIMDs per CU: 0 2022-11-23T01:36:00.8991514Z Shader Engines: 0 2022-11-23T01:36:00.8992130Z Shader Arrs. per Eng.: 0 2022-11-23T01:36:00.8992771Z WatchPts on Addr. Ranges:1 2022-11-23T01:36:00.8993321Z Features: None 2022-11-23T01:36:00.8993828Z Pool Info: 2022-11-23T01:36:00.8994327Z Pool 1 2022-11-23T01:36:00.8994948Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:36:00.8995622Z Size: 264208872(0xfbf81e8) KB 2022-11-23T01:36:00.8996380Z Allocatable: TRUE 2022-11-23T01:36:00.8997018Z Alloc Granule: 4KB 2022-11-23T01:36:00.8997649Z Alloc Alignment: 4KB 2022-11-23T01:36:00.8998299Z Accessible by all: TRUE 2022-11-23T01:36:00.8998864Z Pool 2 2022-11-23T01:36:00.8999483Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:36:00.9000167Z Size: 264208872(0xfbf81e8) KB 2022-11-23T01:36:00.9000801Z Allocatable: TRUE 2022-11-23T01:36:00.9001414Z Alloc Granule: 4KB 2022-11-23T01:36:00.9002045Z Alloc Alignment: 4KB 2022-11-23T01:36:00.9002697Z Accessible by all: TRUE 2022-11-23T01:36:00.9003252Z Pool 3 2022-11-23T01:36:00.9003865Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:36:00.9004518Z Size: 264208872(0xfbf81e8) KB 2022-11-23T01:36:00.9005124Z Allocatable: TRUE 2022-11-23T01:36:00.9005757Z Alloc Granule: 4KB 2022-11-23T01:36:00.9006394Z Alloc Alignment: 4KB 2022-11-23T01:36:00.9007152Z Accessible by all: TRUE 2022-11-23T01:36:00.9007712Z ISA Info: 2022-11-23T01:36:00.9008193Z ******* 2022-11-23T01:36:00.9008638Z Agent 3 2022-11-23T01:36:00.9009089Z ******* 2022-11-23T01:36:00.9009633Z Name: gfx906 2022-11-23T01:36:00.9010486Z Uuid: GPU-17ae48c172fd5d44 2022-11-23T01:36:00.9011122Z Marketing Name: 2022-11-23T01:36:00.9011741Z Vendor Name: AMD 2022-11-23T01:36:00.9012356Z Feature: KERNEL_DISPATCH 2022-11-23T01:36:00.9013003Z Profile: BASE_PROFILE 2022-11-23T01:36:00.9013638Z Float Round Mode: NEAR 2022-11-23T01:36:00.9014264Z Max Queue Number: 128(0x80) 2022-11-23T01:36:00.9014883Z Queue Min Size: 64(0x40) 2022-11-23T01:36:00.9015483Z Queue Max Size: 131072(0x20000) 2022-11-23T01:36:00.9016071Z Queue Type: MULTI 2022-11-23T01:36:00.9016747Z Node: 2 2022-11-23T01:36:00.9017349Z Device Type: GPU 2022-11-23T01:36:00.9017897Z Cache Info: 2022-11-23T01:36:00.9018457Z L1: 16(0x10) KB 2022-11-23T01:36:00.9019055Z Chip ID: 26273(0x66a1) 2022-11-23T01:36:00.9019667Z Cacheline Size: 64(0x40) 2022-11-23T01:36:00.9020264Z Max Clock Freq. (MHz): 1725 2022-11-23T01:36:00.9020845Z BDFID: 16128 2022-11-23T01:36:00.9021450Z Internal Node ID: 2 2022-11-23T01:36:00.9022052Z Compute Unit: 60 2022-11-23T01:36:00.9022644Z SIMDs per CU: 4 2022-11-23T01:36:00.9023240Z Shader Engines: 4 2022-11-23T01:36:00.9023847Z Shader Arrs. per Eng.: 1 2022-11-23T01:36:00.9024486Z WatchPts on Addr. Ranges:4 2022-11-23T01:36:00.9025114Z Features: KERNEL_DISPATCH 2022-11-23T01:36:00.9025735Z Fast F16 Operation: TRUE 2022-11-23T01:36:00.9026357Z Wavefront Size: 64(0x40) 2022-11-23T01:36:00.9026970Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:36:00.9027560Z Workgroup Max Size per Dimension: 2022-11-23T01:36:00.9028165Z x 1024(0x400) 2022-11-23T01:36:00.9028752Z y 1024(0x400) 2022-11-23T01:36:00.9029323Z z 1024(0x400) 2022-11-23T01:36:00.9029936Z Max Waves Per CU: 40(0x28) 2022-11-23T01:36:00.9030781Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:36:00.9031412Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:36:00.9032005Z Grid Max Size per Dimension: 2022-11-23T01:36:00.9032595Z x 4294967295(0xffffffff) 2022-11-23T01:36:00.9033193Z y 4294967295(0xffffffff) 2022-11-23T01:36:00.9033790Z z 4294967295(0xffffffff) 2022-11-23T01:36:00.9034417Z Max fbarriers/Workgrp: 32 2022-11-23T01:36:00.9034974Z Pool Info: 2022-11-23T01:36:00.9035455Z Pool 1 2022-11-23T01:36:00.9036332Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:36:00.9037014Z Size: 16760832(0xffc000) KB 2022-11-23T01:36:00.9037652Z Allocatable: TRUE 2022-11-23T01:36:00.9038303Z Alloc Granule: 4KB 2022-11-23T01:36:00.9038941Z Alloc Alignment: 4KB 2022-11-23T01:36:00.9039567Z Accessible by all: FALSE 2022-11-23T01:36:00.9040140Z Pool 2 2022-11-23T01:36:00.9040679Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:36:00.9041020Z Size: 16760832(0xffc000) KB 2022-11-23T01:36:00.9041353Z Allocatable: TRUE 2022-11-23T01:36:00.9041681Z Alloc Granule: 4KB 2022-11-23T01:36:00.9042005Z Alloc Alignment: 4KB 2022-11-23T01:36:00.9042340Z Accessible by all: FALSE 2022-11-23T01:36:00.9042634Z Pool 3 2022-11-23T01:36:00.9042930Z Segment: GROUP 2022-11-23T01:36:00.9043306Z Size: 64(0x40) KB 2022-11-23T01:36:00.9043631Z Allocatable: FALSE 2022-11-23T01:36:00.9043946Z Alloc Granule: 0KB 2022-11-23T01:36:00.9044271Z Alloc Alignment: 0KB 2022-11-23T01:36:00.9044609Z Accessible by all: FALSE 2022-11-23T01:36:00.9044902Z ISA Info: 2022-11-23T01:36:00.9045162Z ISA 1 2022-11-23T01:36:00.9045626Z Name: amdgcn-amd-amdhsa--gfx906:sramecc-:xnack- 2022-11-23T01:36:00.9046019Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:36:00.9046381Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:36:00.9046734Z Default Rounding Mode: NEAR 2022-11-23T01:36:00.9047085Z Default Rounding Mode: NEAR 2022-11-23T01:36:00.9047414Z Fast f16: TRUE 2022-11-23T01:36:00.9047741Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:36:00.9048061Z Workgroup Max Size per Dimension: 2022-11-23T01:36:00.9048368Z x 1024(0x400) 2022-11-23T01:36:00.9048690Z y 1024(0x400) 2022-11-23T01:36:00.9049001Z z 1024(0x400) 2022-11-23T01:36:00.9049330Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:36:00.9049648Z Grid Max Size per Dimension: 2022-11-23T01:36:00.9049964Z x 4294967295(0xffffffff) 2022-11-23T01:36:00.9050274Z y 4294967295(0xffffffff) 2022-11-23T01:36:00.9050605Z z 4294967295(0xffffffff) 2022-11-23T01:36:00.9050937Z FBarrier Max Size: 32 2022-11-23T01:36:00.9051222Z ******* 2022-11-23T01:36:00.9051471Z Agent 4 2022-11-23T01:36:00.9051718Z ******* 2022-11-23T01:36:00.9051988Z Name: gfx906 2022-11-23T01:36:00.9052420Z Uuid: GPU-17dc608172fd5d44 2022-11-23T01:36:00.9052759Z Marketing Name: 2022-11-23T01:36:00.9053081Z Vendor Name: AMD 2022-11-23T01:36:00.9053533Z Feature: KERNEL_DISPATCH 2022-11-23T01:36:00.9053871Z Profile: BASE_PROFILE 2022-11-23T01:36:00.9054186Z Float Round Mode: NEAR 2022-11-23T01:36:00.9054513Z Max Queue Number: 128(0x80) 2022-11-23T01:36:00.9054838Z Queue Min Size: 64(0x40) 2022-11-23T01:36:00.9055162Z Queue Max Size: 131072(0x20000) 2022-11-23T01:36:00.9055482Z Queue Type: MULTI 2022-11-23T01:36:00.9055794Z Node: 3 2022-11-23T01:36:00.9056098Z Device Type: GPU 2022-11-23T01:36:00.9056367Z Cache Info: 2022-11-23T01:36:00.9056666Z L1: 16(0x10) KB 2022-11-23T01:36:00.9056981Z Chip ID: 26273(0x66a1) 2022-11-23T01:36:00.9057304Z Cacheline Size: 64(0x40) 2022-11-23T01:36:00.9057631Z Max Clock Freq. (MHz): 1725 2022-11-23T01:36:00.9057941Z BDFID: 17152 2022-11-23T01:36:00.9058284Z Internal Node ID: 3 2022-11-23T01:36:00.9058598Z Compute Unit: 60 2022-11-23T01:36:00.9058915Z SIMDs per CU: 4 2022-11-23T01:36:00.9059234Z Shader Engines: 4 2022-11-23T01:36:00.9059555Z Shader Arrs. per Eng.: 1 2022-11-23T01:36:00.9059888Z WatchPts on Addr. Ranges:4 2022-11-23T01:36:00.9060201Z Features: KERNEL_DISPATCH 2022-11-23T01:36:00.9060528Z Fast F16 Operation: TRUE 2022-11-23T01:36:00.9060855Z Wavefront Size: 64(0x40) 2022-11-23T01:36:00.9061182Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:36:00.9061497Z Workgroup Max Size per Dimension: 2022-11-23T01:36:00.9061814Z x 1024(0x400) 2022-11-23T01:36:00.9062104Z y 1024(0x400) 2022-11-23T01:36:00.9062401Z z 1024(0x400) 2022-11-23T01:36:00.9062712Z Max Waves Per CU: 40(0x28) 2022-11-23T01:36:00.9063145Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:36:00.9063482Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:36:00.9063797Z Grid Max Size per Dimension: 2022-11-23T01:36:00.9064090Z x 4294967295(0xffffffff) 2022-11-23T01:36:00.9064409Z y 4294967295(0xffffffff) 2022-11-23T01:36:00.9064724Z z 4294967295(0xffffffff) 2022-11-23T01:36:00.9065052Z Max fbarriers/Workgrp: 32 2022-11-23T01:36:00.9065350Z Pool Info: 2022-11-23T01:36:00.9065614Z Pool 1 2022-11-23T01:36:00.9065921Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:36:00.9066268Z Size: 16760832(0xffc000) KB 2022-11-23T01:36:00.9066603Z Allocatable: TRUE 2022-11-23T01:36:00.9066936Z Alloc Granule: 4KB 2022-11-23T01:36:00.9067270Z Alloc Alignment: 4KB 2022-11-23T01:36:00.9067608Z Accessible by all: FALSE 2022-11-23T01:36:00.9067903Z Pool 2 2022-11-23T01:36:00.9068202Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:36:00.9068594Z Size: 16760832(0xffc000) KB 2022-11-23T01:36:00.9068922Z Allocatable: TRUE 2022-11-23T01:36:00.9069257Z Alloc Granule: 4KB 2022-11-23T01:36:00.9069586Z Alloc Alignment: 4KB 2022-11-23T01:36:00.9069923Z Accessible by all: FALSE 2022-11-23T01:36:00.9070197Z Pool 3 2022-11-23T01:36:00.9070495Z Segment: GROUP 2022-11-23T01:36:00.9070815Z Size: 64(0x40) KB 2022-11-23T01:36:00.9071144Z Allocatable: FALSE 2022-11-23T01:36:00.9071474Z Alloc Granule: 0KB 2022-11-23T01:36:00.9071801Z Alloc Alignment: 0KB 2022-11-23T01:36:00.9072121Z Accessible by all: FALSE 2022-11-23T01:36:00.9072414Z ISA Info: 2022-11-23T01:36:00.9072674Z ISA 1 2022-11-23T01:36:00.9073129Z Name: amdgcn-amd-amdhsa--gfx906:sramecc-:xnack- 2022-11-23T01:36:00.9073579Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:36:00.9073943Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:36:00.9074294Z Default Rounding Mode: NEAR 2022-11-23T01:36:00.9074625Z Default Rounding Mode: NEAR 2022-11-23T01:36:00.9074955Z Fast f16: TRUE 2022-11-23T01:36:00.9075284Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:36:00.9075608Z Workgroup Max Size per Dimension: 2022-11-23T01:36:00.9075934Z x 1024(0x400) 2022-11-23T01:36:00.9076318Z y 1024(0x400) 2022-11-23T01:36:00.9076613Z z 1024(0x400) 2022-11-23T01:36:00.9076938Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:36:00.9077256Z Grid Max Size per Dimension: 2022-11-23T01:36:00.9077568Z x 4294967295(0xffffffff) 2022-11-23T01:36:00.9077890Z y 4294967295(0xffffffff) 2022-11-23T01:36:00.9078212Z z 4294967295(0xffffffff) 2022-11-23T01:36:00.9078523Z FBarrier Max Size: 32 2022-11-23T01:36:00.9078814Z *** Done *** 2022-11-23T01:36:00.9098872Z + rocminfo 2022-11-23T01:36:00.9099506Z + grep -E 'Name:.*\sgfx|Marketing' 2022-11-23T01:36:00.9728266Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:36:00.9728711Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:36:00.9730989Z Name: gfx906 2022-11-23T01:36:00.9732585Z Marketing Name: 2022-11-23T01:36:00.9733413Z Name: gfx906 2022-11-23T01:36:00.9734203Z Marketing Name: 2022-11-23T01:36:00.9901519Z + [[ linux-focal-rocm5.2-py3.8 != *-bazel-* ]] 2022-11-23T01:36:00.9903284Z + pip_install --user ninja==1.10.2 2022-11-23T01:36:00.9904395Z + pip install --progress-bar off --user ninja==1.10.2 2022-11-23T01:36:01.5204369Z Collecting ninja==1.10.2 2022-11-23T01:36:01.5552770Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2022-11-23T01:36:02.1873833Z Installing collected packages: ninja 2022-11-23T01:36:02.1954286Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-11-23T01:36:02.1957086Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-11-23T01:36:02.1983808Z Successfully installed ninja-1.10.2 2022-11-23T01:36:02.2579876Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:36:02.2583105Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:36:02.2584822Z + [[ linux-focal-rocm5.2-py3.8 == *asan* ]] 2022-11-23T01:36:02.2585896Z + [[ linux-focal-rocm5.2-py3.8 == *-tsan* ]] 2022-11-23T01:36:02.2586762Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-11-23T01:36:02.2587612Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2022-11-23T01:36:02.2604241Z + [[ linux-focal-rocm5.2-py3.8 == *tbb* ]] 2022-11-23T01:36:02.2643359Z + [[ linux-focal-rocm5.2-py3.8 == *libtorch* ]] 2022-11-23T01:36:02.2645248Z + [[ linux-focal-rocm5.2-py3.8 == *-bazel-* ]] 2022-11-23T01:36:02.2647671Z + [[ linux-focal-rocm5.2-py3.8 == *-tsan* ]] 2022-11-23T01:36:02.2648444Z + cd test 2022-11-23T01:36:02.2649451Z + python -c 'import torch; print(torch.__config__.show())' 2022-11-23T01:36:03.6457469Z PyTorch built with: 2022-11-23T01:36:03.6459386Z - GCC 9.4 2022-11-23T01:36:03.6460775Z - C++ Version: 201402 2022-11-23T01:36:03.6462262Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:36:03.6463818Z - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:36:03.6464888Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:36:03.6465901Z - LAPACK is enabled (usually provided by MKL) 2022-11-23T01:36:03.6466780Z - NNPACK is enabled 2022-11-23T01:36:03.6467625Z - CPU capability usage: AVX2 2022-11-23T01:36:03.6468473Z - HIP Runtime 5.2.21151 2022-11-23T01:36:03.6469235Z - MIOpen 2.17.0 2022-11-23T01:36:03.6469941Z - Magma 2.6.1 2022-11-23T01:36:03.6479318Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, 2022-11-23T01:36:03.6533100Z 2022-11-23T01:36:04.9738063Z + cd test 2022-11-23T01:36:04.9740214Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2022-11-23T01:36:06.2501148Z ATen/Parallel: 2022-11-23T01:36:06.2528141Z at::get_num_threads() : 28 2022-11-23T01:36:06.2528797Z at::get_num_interop_threads() : 28 2022-11-23T01:36:06.2529344Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:36:06.2529857Z omp_get_max_threads() : 28 2022-11-23T01:36:06.2531108Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:36:06.2532238Z mkl_get_max_threads() : 28 2022-11-23T01:36:06.2533083Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:36:06.2533758Z std::thread::hardware_concurrency() : 56 2022-11-23T01:36:06.2534309Z Environment variables: 2022-11-23T01:36:06.2534804Z OMP_NUM_THREADS : [not set] 2022-11-23T01:36:06.2535355Z MKL_NUM_THREADS : [not set] 2022-11-23T01:36:06.2535870Z ATen parallel backend: OpenMP 2022-11-23T01:36:06.2536192Z 2022-11-23T01:36:07.5437525Z + [[ distributed == *backward* ]] 2022-11-23T01:36:07.5439116Z + [[ distributed == *xla* ]] 2022-11-23T01:36:07.5440162Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-11-23T01:36:07.5441502Z + [[ linux-focal-rocm5.2-py3.8 == *libtorch* ]] 2022-11-23T01:36:07.5442331Z + [[ distributed == distributed ]] 2022-11-23T01:36:07.5442985Z + install_filelock 2022-11-23T01:36:07.5443621Z + pip_install filelock 2022-11-23T01:36:07.5444587Z + pip install --progress-bar off filelock 2022-11-23T01:36:08.0291087Z Collecting filelock 2022-11-23T01:36:08.0555032Z Downloading filelock-3.8.0-py3-none-any.whl (10 kB) 2022-11-23T01:36:08.7264093Z Installing collected packages: filelock 2022-11-23T01:36:08.7508318Z Successfully installed filelock-3.8.0 2022-11-23T01:36:08.8250106Z + install_triton 2022-11-23T01:36:08.8251365Z + local commit 2022-11-23T01:36:08.8251998Z + [[ distributed == *rocm* ]] 2022-11-23T01:36:08.8256305Z ++ get_pinned_commit triton 2022-11-23T01:36:08.8257834Z ++ cat .github/ci_commit_pins/triton.txt 2022-11-23T01:36:08.8289071Z + commit=0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:36:08.8291847Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:36:08.8293839Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:36:09.2508913Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:36:09.2511696Z Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-ljyrq49x 2022-11-23T01:36:09.2571844Z Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-ljyrq49x 2022-11-23T01:36:11.3051333Z Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331' 2022-11-23T01:36:11.3124953Z Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:36:12.0695466Z Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:36:12.6568067Z Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:36:12.6573479Z Running command git submodule update --init --recursive -q 2022-11-23T01:36:13.8352522Z Preparing metadata (setup.py) ... [?25l- done 2022-11-23T01:36:14.0592662Z [?25hCollecting cmake 2022-11-23T01:36:14.0940342Z Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB) 2022-11-23T01:36:14.6295306Z Requirement already satisfied: filelock in /opt/conda/lib/python3.8/site-packages (from triton==2.0.0) (3.8.0) 2022-11-23T01:36:14.6297487Z Requirement already satisfied: torch in /opt/conda/lib/python3.8/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385) 2022-11-23T01:36:14.6543646Z Requirement already satisfied: sympy in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (1.11.1) 2022-11-23T01:36:14.6545626Z Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (2.6.3) 2022-11-23T01:36:14.6548875Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (4.4.0) 2022-11-23T01:36:14.6744390Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy->torch->triton==2.0.0) (1.2.1) 2022-11-23T01:36:14.6928468Z Building wheels for collected packages: triton 2022-11-23T01:37:19.5574165Z Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / done 2022-11-23T01:37:19.6048383Z [?25h Created wheel for triton: filename=triton-2.0.0-cp38-cp38-linux_x86_64.whl size=15414539 sha256=74849060bfa87492b4bd6ee01c7b59c39c501a7ee3f65b69a2bf35310f45301f 2022-11-23T01:37:19.6052177Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/c0/c0/56/bdb2859a55c7764d4e97889d26a8a05b683ef97fe9b1aa7dec 2022-11-23T01:37:19.6088554Z Successfully built triton 2022-11-23T01:37:20.2405723Z Installing collected packages: cmake, triton 2022-11-23T01:37:21.2559763Z Successfully installed cmake-3.25.0 triton-2.0.0 2022-11-23T01:37:21.3558994Z + pip_install --user jinja2 2022-11-23T01:37:21.3560152Z + pip install --progress-bar off --user jinja2 2022-11-23T01:37:21.9061664Z Collecting jinja2 2022-11-23T01:37:22.0849379Z Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) 2022-11-23T01:37:22.1365730Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.8/site-packages (from jinja2) (2.1.1) 2022-11-23T01:37:22.7793887Z Installing collected packages: jinja2 2022-11-23T01:37:22.8510595Z Successfully installed jinja2-3.1.2 2022-11-23T01:37:22.9315018Z + test_distributed 2022-11-23T01:37:22.9316659Z + echo 'Testing distributed python tests' 2022-11-23T01:37:22.9317497Z Testing distributed python tests 2022-11-23T01:37:22.9319021Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose 2022-11-23T01:37:25.2783928Z Ignoring disabled issues: [] 2022-11-23T01:37:25.2937394Z Excluding distributed/rpc/test_faulty_agent on ROCm 2022-11-23T01:37:25.2938453Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm 2022-11-23T01:37:25.2939440Z Excluding distributed/rpc/test_share_memory on ROCm 2022-11-23T01:37:25.2940419Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm 2022-11-23T01:37:25.2941502Z Excluding distributed/_shard/sharding_plan/test_sharding_plan on ROCm 2022-11-23T01:37:25.2942625Z Excluding distributed/_shard/sharded_tensor/test_megatron_prototype on ROCm 2022-11-23T01:37:25.2943739Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor on ROCm 2022-11-23T01:37:25.2946016Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor_reshard on ROCm 2022-11-23T01:37:25.2947215Z Excluding distributed/_shard/sharded_tensor/ops/test_chunk on ROCm 2022-11-23T01:37:25.2948405Z Excluding distributed/_shard/sharded_tensor/ops/test_elementwise_ops on ROCm 2022-11-23T01:37:25.2950109Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding on ROCm 2022-11-23T01:37:25.2951842Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding_bag on ROCm 2022-11-23T01:37:25.2953001Z Excluding distributed/_shard/sharded_tensor/ops/test_binary_cmp on ROCm 2022-11-23T01:37:25.2954052Z Excluding distributed/_shard/sharded_tensor/ops/test_init on ROCm 2022-11-23T01:37:25.2955134Z Excluding distributed/_shard/sharded_tensor/ops/test_linear on ROCm 2022-11-23T01:37:25.2956401Z Excluding distributed/_shard/sharded_tensor/ops/test_math_ops on ROCm 2022-11-23T01:37:25.2957488Z Excluding distributed/_shard/sharded_tensor/ops/test_matrix_ops on ROCm 2022-11-23T01:37:25.2958572Z Excluding distributed/_shard/sharded_tensor/ops/test_softmax on ROCm 2022-11-23T01:37:25.2959636Z Excluding distributed/_shard/sharded_optim/test_sharded_optim on ROCm 2022-11-23T01:37:25.2960663Z Excluding distributed/_shard/test_partial_tensor on ROCm 2022-11-23T01:37:25.2961656Z Excluding distributed/_shard/test_replicated_tensor on ROCm 2022-11-23T01:37:25.3018494Z ##[warning] Gathered no stats from artifacts. Proceeding with default sharding plan. 2022-11-23T01:37:25.3020565Z Selected tests: 2022-11-23T01:37:25.3021360Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:25.3022234Z distributed/test_distributed_spawn 2022-11-23T01:37:25.3023397Z distributed/test_pg_wrapper 2022-11-23T01:37:25.3024135Z distributed/test_multi_threaded_pg 2022-11-23T01:37:25.3024908Z distributed/test_dynamo_distributed 2022-11-23T01:37:25.3025640Z distributed/test_c10d_spawn_ucc 2022-11-23T01:37:25.3026375Z distributed/test_c10d_spawn_gloo 2022-11-23T01:37:25.3027248Z distributed/test_c10d_object_collectives 2022-11-23T01:37:25.3027998Z distributed/test_c10d_gloo 2022-11-23T01:37:25.3028695Z distributed/test_c10d_common 2022-11-23T01:37:25.3029493Z distributed/pipeline/sync/test_transparency 2022-11-23T01:37:25.3030344Z distributed/pipeline/sync/test_pipeline 2022-11-23T01:37:25.3031130Z distributed/pipeline/sync/test_phony 2022-11-23T01:37:25.3031923Z distributed/pipeline/sync/test_inplace 2022-11-23T01:37:25.3032790Z distributed/pipeline/sync/test_deferred_batch_norm 2022-11-23T01:37:25.3033673Z distributed/pipeline/sync/test_checkpoint 2022-11-23T01:37:25.3034482Z distributed/pipeline/sync/test_balance 2022-11-23T01:37:25.3035319Z distributed/pipeline/sync/skip/test_tracker 2022-11-23T01:37:25.3036444Z distributed/pipeline/sync/skip/test_portal 2022-11-23T01:37:25.3037348Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-11-23T01:37:25.3038224Z distributed/pipeline/sync/skip/test_api 2022-11-23T01:37:25.3039096Z distributed/optim/test_apply_optimizer_in_backward 2022-11-23T01:37:25.3040063Z distributed/fsdp/test_wrap 2022-11-23T01:37:25.3040784Z distributed/fsdp/test_shard_utils 2022-11-23T01:37:25.3041532Z distributed/fsdp/test_fsdp_uneven 2022-11-23T01:37:25.3042332Z distributed/fsdp/test_fsdp_tp_integration 2022-11-23T01:37:25.3043141Z distributed/fsdp/test_fsdp_state_dict 2022-11-23T01:37:25.3043924Z distributed/fsdp/test_fsdp_pure_fp16 2022-11-23T01:37:25.3044707Z distributed/fsdp/test_fsdp_optim_state 2022-11-23T01:37:25.3045515Z distributed/fsdp/test_fsdp_multiple_forward 2022-11-23T01:37:25.3046310Z distributed/fsdp/test_fsdp_misc 2022-11-23T01:37:25.3047052Z distributed/fsdp/test_fsdp_memory 2022-11-23T01:37:25.3047861Z distributed/fsdp/test_fsdp_ignored_modules 2022-11-23T01:37:25.3048632Z distributed/fsdp/test_fsdp_fx 2022-11-23T01:37:25.3049420Z distributed/fsdp/test_fsdp_flatten_params 2022-11-23T01:37:25.3050179Z distributed/fsdp/test_fsdp_core 2022-11-23T01:37:25.3050913Z distributed/fsdp/test_fsdp_comm 2022-11-23T01:37:25.3051684Z distributed/fsdp/test_fsdp_checkpoint 2022-11-23T01:37:25.3052529Z distributed/fsdp/test_distributed_checkpoint 2022-11-23T01:37:25.3053348Z distributed/elastic/utils/util_test 2022-11-23T01:37:25.3054373Z distributed/elastic/utils/distributed_test 2022-11-23T01:37:25.3055973Z distributed/elastic/timer/local_timer_example 2022-11-23T01:37:25.3057216Z distributed/elastic/multiprocessing/api_test 2022-11-23T01:37:25.3058035Z distributed/elastic/events/lib_test 2022-11-23T01:37:25.3058809Z distributed/checkpoint/test_traverse 2022-11-23T01:37:25.3059688Z distributed/checkpoint/test_file_system_checkpoint_cpu 2022-11-23T01:37:25.3060574Z distributed/checkpoint/test_dedup_tensors 2022-11-23T01:37:25.3061368Z distributed/algorithms/test_join 2022-11-23T01:37:25.3062115Z distributed/_tensor/test_view_ops 2022-11-23T01:37:25.3062869Z distributed/_tensor/test_tensor_ops 2022-11-23T01:37:25.3063650Z distributed/_tensor/test_pointwise_ops 2022-11-23T01:37:25.3064414Z distributed/_tensor/test_math_ops 2022-11-23T01:37:25.3065156Z distributed/_tensor/test_device_mesh 2022-11-23T01:37:25.3065894Z distributed/_tensor/test_api 2022-11-23T01:37:25.3066676Z distributed/_tensor/parallel/test_tp_style 2022-11-23T01:37:25.3067555Z distributed/_tensor/parallel/test_parallelize_api 2022-11-23T01:37:25.3068379Z distributed/_shard/test_sharder 2022-11-23T01:37:25.3069296Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-11-23T01:37:25.3070244Z distributed/_composable/test_fully_shard 2022-11-23T01:37:25.3071067Z distributed/_composable/test_checkpoint 2022-11-23T01:37:25.3071923Z Prioritized test from test file changes. 2022-11-23T01:37:25.3072657Z reordering tests for PR: 2022-11-23T01:37:25.3073490Z prioritized: [] 2022-11-23T01:37:25.3082656Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/test_pg_wrapper', 'distributed/test_multi_threaded_pg', 'distributed/test_dynamo_distributed', 'distributed/test_c10d_spawn_ucc', 'distributed/test_c10d_spawn_gloo', 'distributed/test_c10d_object_collectives', 'distributed/test_c10d_gloo', 'distributed/test_c10d_common', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_api', 'distributed/optim/test_apply_optimizer_in_backward', 'distributed/fsdp/test_wrap', 'distributed/fsdp/test_shard_utils', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_fsdp_tp_integration', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/fsdp/test_fsdp_optim_state', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/fsdp/test_fsdp_misc', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/fsdp/test_fsdp_fx', 'distributed/fsdp/test_fsdp_flatten_params', 'distributed/fsdp/test_fsdp_core', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/elastic/utils/util_test', 'distributed/elastic/utils/distributed_test', 'distributed/elastic/timer/local_timer_example', 'distributed/elastic/multiprocessing/api_test', 'distributed/elastic/events/lib_test', 'distributed/checkpoint/test_traverse', 'distributed/checkpoint/test_file_system_checkpoint_cpu', 'distributed/checkpoint/test_dedup_tensors', 'distributed/algorithms/test_join', 'distributed/_tensor/test_view_ops', 'distributed/_tensor/test_tensor_ops', 'distributed/_tensor/test_pointwise_ops', 'distributed/_tensor/test_math_ops', 'distributed/_tensor/test_device_mesh', 'distributed/_tensor/test_api', 'distributed/_tensor/parallel/test_tp_style', 'distributed/_tensor/parallel/test_parallelize_api', 'distributed/_shard/test_sharder', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_composable/test_fully_shard', 'distributed/_composable/test_checkpoint'] 2022-11-23T01:37:25.3086315Z 2022-11-23T01:37:25.3086906Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2022-11-23T01:37:25.4945656Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2022-11-23T01:37:25.5560087Z parallel (file granularity) tests: 2022-11-23T01:37:25.5561333Z 2022-11-23T01:37:25.5562251Z serial (file granularity) tests: 2022-11-23T01:37:25.5563763Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:25.5565049Z distributed/test_distributed_spawn 2022-11-23T01:37:25.5566505Z distributed/test_pg_wrapper 2022-11-23T01:37:25.5567650Z distributed/test_multi_threaded_pg 2022-11-23T01:37:25.5568415Z distributed/test_dynamo_distributed 2022-11-23T01:37:25.5569185Z distributed/test_c10d_spawn_ucc 2022-11-23T01:37:25.5569934Z distributed/test_c10d_spawn_gloo 2022-11-23T01:37:25.5570721Z distributed/test_c10d_object_collectives 2022-11-23T01:37:25.5571554Z distributed/test_c10d_gloo 2022-11-23T01:37:25.5572266Z distributed/test_c10d_common 2022-11-23T01:37:25.5573071Z distributed/pipeline/sync/test_transparency 2022-11-23T01:37:25.5573907Z distributed/pipeline/sync/test_pipeline 2022-11-23T01:37:25.5574722Z distributed/pipeline/sync/test_phony 2022-11-23T01:37:25.5575528Z distributed/pipeline/sync/test_inplace 2022-11-23T01:37:25.5576831Z distributed/pipeline/sync/test_deferred_batch_norm 2022-11-23T01:37:25.5577730Z distributed/pipeline/sync/test_checkpoint 2022-11-23T01:37:25.5578565Z distributed/pipeline/sync/test_balance 2022-11-23T01:37:25.5579382Z distributed/pipeline/sync/skip/test_tracker 2022-11-23T01:37:25.5580255Z distributed/pipeline/sync/skip/test_portal 2022-11-23T01:37:25.5581178Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-11-23T01:37:25.5582073Z distributed/pipeline/sync/skip/test_api 2022-11-23T01:37:25.5582960Z distributed/optim/test_apply_optimizer_in_backward 2022-11-23T01:37:25.5583773Z distributed/fsdp/test_wrap 2022-11-23T01:37:25.5584488Z distributed/fsdp/test_shard_utils 2022-11-23T01:37:25.5585260Z distributed/fsdp/test_fsdp_uneven 2022-11-23T01:37:25.5586069Z distributed/fsdp/test_fsdp_tp_integration 2022-11-23T01:37:25.5586888Z distributed/fsdp/test_fsdp_state_dict 2022-11-23T01:37:25.5587681Z distributed/fsdp/test_fsdp_pure_fp16 2022-11-23T01:37:25.5588459Z distributed/fsdp/test_fsdp_optim_state 2022-11-23T01:37:25.5589296Z distributed/fsdp/test_fsdp_multiple_forward 2022-11-23T01:37:25.5590095Z distributed/fsdp/test_fsdp_misc 2022-11-23T01:37:25.5590853Z distributed/fsdp/test_fsdp_memory 2022-11-23T01:37:25.5591656Z distributed/fsdp/test_fsdp_ignored_modules 2022-11-23T01:37:25.5592589Z distributed/fsdp/test_fsdp_fx 2022-11-23T01:37:25.5593360Z distributed/fsdp/test_fsdp_flatten_params 2022-11-23T01:37:25.5594149Z distributed/fsdp/test_fsdp_core 2022-11-23T01:37:25.5594887Z distributed/fsdp/test_fsdp_comm 2022-11-23T01:37:25.5595658Z distributed/fsdp/test_fsdp_checkpoint 2022-11-23T01:37:25.5596729Z distributed/fsdp/test_distributed_checkpoint 2022-11-23T01:37:25.5597564Z distributed/elastic/utils/util_test 2022-11-23T01:37:25.5598367Z distributed/elastic/utils/distributed_test 2022-11-23T01:37:25.5599242Z distributed/elastic/timer/local_timer_example 2022-11-23T01:37:25.5600133Z distributed/elastic/multiprocessing/api_test 2022-11-23T01:37:25.5600972Z distributed/elastic/events/lib_test 2022-11-23T01:37:25.5601755Z distributed/checkpoint/test_traverse 2022-11-23T01:37:25.5602626Z distributed/checkpoint/test_file_system_checkpoint_cpu 2022-11-23T01:37:25.5603529Z distributed/checkpoint/test_dedup_tensors 2022-11-23T01:37:25.5604329Z distributed/algorithms/test_join 2022-11-23T01:37:25.5605101Z distributed/_tensor/test_view_ops 2022-11-23T01:37:25.5605871Z distributed/_tensor/test_tensor_ops 2022-11-23T01:37:25.5606668Z distributed/_tensor/test_pointwise_ops 2022-11-23T01:37:25.5607431Z distributed/_tensor/test_math_ops 2022-11-23T01:37:25.5608191Z distributed/_tensor/test_device_mesh 2022-11-23T01:37:25.5608937Z distributed/_tensor/test_api 2022-11-23T01:37:25.5609733Z distributed/_tensor/parallel/test_tp_style 2022-11-23T01:37:25.5610635Z distributed/_tensor/parallel/test_parallelize_api 2022-11-23T01:37:25.5611464Z distributed/_shard/test_sharder 2022-11-23T01:37:25.5612303Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-11-23T01:37:25.5613204Z distributed/_composable/test_fully_shard 2022-11-23T01:37:25.5614034Z distributed/_composable/test_checkpoint 2022-11-23T01:37:27.7200056Z Ignoring disabled issues: [] 2022-11-23T01:37:27.7489381Z Ignoring disabled issues: [] 2022-11-23T01:37:29.2980658Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:37:29.297067] 2022-11-23T01:37:29.3165461Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:37:29.3167183Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2} 2022-11-23T01:37:29.3168519Z Running distributed tests for the test backend with env init_method in shard 1 of 2 2022-11-23T01:37:29.3175529Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:37:29.317164] 2022-11-23T01:37:32.2576755Z 2022-11-23T01:37:32.2578509Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:32.2584567Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_wtvdfxuo) 2022-11-23T01:37:32.2585209Z 2022-11-23T01:37:32.2585415Z 2022-11-23T01:37:32.2585761Z ##[endgroup] 2022-11-23T01:37:32.2586746Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_wtvdfxuo) 2022-11-23T01:37:32.2587216Z 2022-11-23T01:37:32.2602397Z Running distributed tests for the test backend with file init_method in shard 1 of 2 2022-11-23T01:37:32.2615119Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:37:32.260650] 2022-11-23T01:37:35.1502030Z 2022-11-23T01:37:35.1503582Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:35.1506666Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_7_ufbikh) 2022-11-23T01:37:35.1508254Z 2022-11-23T01:37:35.1508737Z 2022-11-23T01:37:35.1509517Z ##[endgroup] 2022-11-23T01:37:35.1511712Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_7_ufbikh) 2022-11-23T01:37:35.1512862Z 2022-11-23T01:37:35.1521844Z Shard 1: nccl should be run in 2 2022-11-23T01:37:35.1522825Z Running distributed tests for the gloo backend with env init_method in shard 1 of 2 2022-11-23T01:37:35.1533550Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:37:35.152928] 2022-11-23T01:38:01.9096494Z 2022-11-23T01:38:01.9104074Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:38:01.9106686Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_wm27g5qr) 2022-11-23T01:38:01.9111819Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-11-23T01:38:01.9115510Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:01.9117533Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:01.9118935Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:01.9120279Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:01.9121610Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:01.9122986Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:01.9124317Z 2022-11-23T01:38:01.9126812Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:01.9129500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9131751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9134147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9135930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9136880Z 2022-11-23T01:38:01.9167129Z Running tests... 2022-11-23T01:38:01.9168587Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9170082Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1240 2022-11-23T01:38:01.9171671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1241 2022-11-23T01:38:01.9173039Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:01.9174905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9176133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9177747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9179385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9180564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:01.9182283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9183451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9185049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9186302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9187490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:01.9188812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:38:01.9190153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:38:01.9192000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:01.9193897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:01.9194944Z ok (3.609s) 2022-11-23T01:38:01.9195316Z 2022-11-23T01:38:01.9196180Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9197044Z Ran 1 test in 3.609s 2022-11-23T01:38:01.9197445Z 2022-11-23T01:38:01.9197653Z OK 2022-11-23T01:38:01.9197978Z 2022-11-23T01:38:01.9198272Z Generating XML reports... 2022-11-23T01:38:01.9200167Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013738.xml 2022-11-23T01:38:01.9202320Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:01.9204245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9205441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9207038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9208293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9208891Z 2022-11-23T01:38:01.9209140Z Running tests... 2022-11-23T01:38:01.9210251Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9211631Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1423 2022-11-23T01:38:01.9213214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1424 2022-11-23T01:38:01.9214574Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:01.9216398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9217604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9219200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9220451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9221619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:01.9223286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9224483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9226063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9227427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9228596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:01.9229897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:38:01.9231226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:38:01.9233030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:01.9234954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:01.9235981Z ok (3.606s) 2022-11-23T01:38:01.9236484Z 2022-11-23T01:38:01.9237233Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9238094Z Ran 1 test in 3.607s 2022-11-23T01:38:01.9238490Z 2022-11-23T01:38:01.9238710Z OK 2022-11-23T01:38:01.9239031Z 2022-11-23T01:38:01.9239326Z Generating XML reports... 2022-11-23T01:38:01.9240494Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013744.xml 2022-11-23T01:38:01.9241352Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:01.9242094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9242578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9243221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9243725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9243964Z 2022-11-23T01:38:01.9244071Z Running tests... 2022-11-23T01:38:01.9244525Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9245018Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2022-11-23T01:38:01.9245314Z 2022-11-23T01:38:01.9245603Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9245951Z Ran 1 test in 0.001s 2022-11-23T01:38:01.9246116Z 2022-11-23T01:38:01.9246221Z OK (skipped=1) 2022-11-23T01:38:01.9246378Z 2022-11-23T01:38:01.9246499Z Generating XML reports... 2022-11-23T01:38:01.9247233Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013751.xml 2022-11-23T01:38:01.9248156Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:01.9248874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9249357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9249990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9250493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9250734Z 2022-11-23T01:38:01.9250840Z Running tests... 2022-11-23T01:38:01.9251298Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9251809Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2022-11-23T01:38:01.9252109Z 2022-11-23T01:38:01.9252396Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9252721Z Ran 1 test in 0.001s 2022-11-23T01:38:01.9252889Z 2022-11-23T01:38:01.9252996Z OK (skipped=1) 2022-11-23T01:38:01.9253154Z 2022-11-23T01:38:01.9253275Z Generating XML reports... 2022-11-23T01:38:01.9254055Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013753.xml 2022-11-23T01:38:01.9254904Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:01.9255642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9256120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9256735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9257238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9257480Z 2022-11-23T01:38:01.9257613Z Running tests... 2022-11-23T01:38:01.9258060Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9258586Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s) 2022-11-23T01:38:01.9258908Z 2022-11-23T01:38:01.9259195Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9259538Z Ran 1 test in 0.001s 2022-11-23T01:38:01.9259704Z 2022-11-23T01:38:01.9259806Z OK (skipped=1) 2022-11-23T01:38:01.9259960Z 2022-11-23T01:38:01.9260062Z Generating XML reports... 2022-11-23T01:38:01.9260794Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013756.xml 2022-11-23T01:38:01.9261646Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:01.9262389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:01.9262873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:01.9263506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:01.9264011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:01.9264250Z 2022-11-23T01:38:01.9264356Z Running tests... 2022-11-23T01:38:01.9264779Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9265305Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s) 2022-11-23T01:38:01.9265619Z 2022-11-23T01:38:01.9265905Z ---------------------------------------------------------------------- 2022-11-23T01:38:01.9266304Z Ran 1 test in 0.001s 2022-11-23T01:38:01.9266468Z 2022-11-23T01:38:01.9266570Z OK (skipped=1) 2022-11-23T01:38:01.9266728Z 2022-11-23T01:38:01.9266848Z Generating XML reports... 2022-11-23T01:38:01.9267578Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013759.xml 2022-11-23T01:38:01.9268008Z 2022-11-23T01:38:01.9268595Z ##[endgroup] 2022-11-23T01:38:01.9269389Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_wm27g5qr) 2022-11-23T01:38:01.9269857Z 2022-11-23T01:38:01.9270079Z Running distributed tests for the gloo backend with file init_method in shard 1 of 2 2022-11-23T01:38:01.9271014Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:38:01.913170] 2022-11-23T01:38:28.5374198Z 2022-11-23T01:38:28.5375386Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:38:28.5382354Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_qcxqht1x) 2022-11-23T01:38:28.5387615Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-11-23T01:38:28.5390402Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:28.5391388Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:28.5392373Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:28.5394039Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:28.5395699Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:28.5398035Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:28.5399020Z 2022-11-23T01:38:28.5400964Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:28.5402927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5404847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5407474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5409791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5410410Z 2022-11-23T01:38:28.5410665Z Running tests... 2022-11-23T01:38:28.5411789Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5413458Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1896 2022-11-23T01:38:28.5414888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1897 2022-11-23T01:38:28.5416251Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:28.5418083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5419278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5420878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5422366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5423545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:28.5425282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5426475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5428057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5429311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5430486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:28.5431790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:38:28.5433096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:38:28.5434950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:28.5437231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:28.5438275Z ok (3.504s) 2022-11-23T01:38:28.5438642Z 2022-11-23T01:38:28.5439395Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5440266Z Ran 1 test in 3.505s 2022-11-23T01:38:28.5440668Z 2022-11-23T01:38:28.5440875Z OK 2022-11-23T01:38:28.5441169Z 2022-11-23T01:38:28.5441475Z Generating XML reports... 2022-11-23T01:38:28.5443340Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013804.xml 2022-11-23T01:38:28.5445504Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:28.5447374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5448598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5450217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5451465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5452159Z 2022-11-23T01:38:28.5452416Z Running tests... 2022-11-23T01:38:28.5453535Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5454903Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2079 2022-11-23T01:38:28.5456306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2080 2022-11-23T01:38:28.5457670Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:28.5459476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5460679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5462286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5463534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5464705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:28.5466392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5467571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5469175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5470587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5471756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:28.5473078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:38:28.5474935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:28.5476510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:38:28.5478341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:28.5479374Z ok (3.504s) 2022-11-23T01:38:28.5479740Z 2022-11-23T01:38:28.5480492Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5481358Z Ran 1 test in 3.505s 2022-11-23T01:38:28.5481754Z 2022-11-23T01:38:28.5481968Z OK 2022-11-23T01:38:28.5482290Z 2022-11-23T01:38:28.5482586Z Generating XML reports... 2022-11-23T01:38:28.5484560Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013811.xml 2022-11-23T01:38:28.5486753Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:28.5488634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5489840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5491446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5492720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5493317Z 2022-11-23T01:38:28.5493567Z Running tests... 2022-11-23T01:38:28.5494690Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5495940Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2022-11-23T01:38:28.5496681Z 2022-11-23T01:38:28.5497415Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5498262Z Ran 1 test in 0.001s 2022-11-23T01:38:28.5498664Z 2022-11-23T01:38:28.5498909Z OK (skipped=1) 2022-11-23T01:38:28.5499284Z 2022-11-23T01:38:28.5499584Z Generating XML reports... 2022-11-23T01:38:28.5501460Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013817.xml 2022-11-23T01:38:28.5503622Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:28.5505460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5506652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5508246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5509512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5510106Z 2022-11-23T01:38:28.5510355Z Running tests... 2022-11-23T01:38:28.5511481Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5512758Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2022-11-23T01:38:28.5513495Z 2022-11-23T01:38:28.5514209Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5515025Z Ran 1 test in 0.001s 2022-11-23T01:38:28.5515425Z 2022-11-23T01:38:28.5533168Z OK (skipped=1) 2022-11-23T01:38:28.5533607Z 2022-11-23T01:38:28.5533759Z Generating XML reports... 2022-11-23T01:38:28.5534630Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013820.xml 2022-11-23T01:38:28.5535514Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:28.5536254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5536754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5537410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5537927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5538168Z 2022-11-23T01:38:28.5538281Z Running tests... 2022-11-23T01:38:28.5538744Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5539288Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s) 2022-11-23T01:38:28.5539611Z 2022-11-23T01:38:28.5539904Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5540231Z Ran 1 test in 0.001s 2022-11-23T01:38:28.5540476Z 2022-11-23T01:38:28.5540588Z OK (skipped=1) 2022-11-23T01:38:28.5540753Z 2022-11-23T01:38:28.5540879Z Generating XML reports... 2022-11-23T01:38:28.5541623Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013823.xml 2022-11-23T01:38:28.5542479Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:28.5543230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:28.5543724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:28.5544351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:28.5544865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:28.5545103Z 2022-11-23T01:38:28.5545222Z Running tests... 2022-11-23T01:38:28.5545684Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5546219Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s) 2022-11-23T01:38:28.5546539Z 2022-11-23T01:38:28.5546832Z ---------------------------------------------------------------------- 2022-11-23T01:38:28.5547184Z Ran 1 test in 0.001s 2022-11-23T01:38:28.5547353Z 2022-11-23T01:38:28.5547438Z OK (skipped=1) 2022-11-23T01:38:28.5547598Z 2022-11-23T01:38:28.5547730Z Generating XML reports... 2022-11-23T01:38:28.5548474Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013825.xml 2022-11-23T01:38:28.5548917Z 2022-11-23T01:38:28.5549343Z ##[endgroup] 2022-11-23T01:38:28.5550147Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_qcxqht1x) 2022-11-23T01:38:28.5550612Z 2022-11-23T01:38:28.5550917Z Running distributed/test_distributed_spawn ... [2022-11-23 01:38:28.540940] 2022-11-23T01:38:28.5587673Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:38:28.5588338Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 1} 2022-11-23T01:38:28.5588917Z Running distributed tests for the test backend with env init_method in shard 1 of 2 2022-11-23T01:38:28.5598283Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:38:28.559085] 2022-11-23T01:38:31.4480030Z 2022-11-23T01:38:31.4481308Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T01:38:31.4483720Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_qtkc9h5u) 2022-11-23T01:38:31.4484613Z 2022-11-23T01:38:31.4484963Z 2022-11-23T01:38:31.4485454Z ##[endgroup] 2022-11-23T01:38:31.4486874Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_qtkc9h5u) 2022-11-23T01:38:31.4487505Z 2022-11-23T01:38:31.4505581Z Running distributed tests for the test backend with file init_method in shard 1 of 2 2022-11-23T01:38:31.4513644Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:38:31.450947] 2022-11-23T01:38:34.4754391Z 2022-11-23T01:38:34.4756787Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T01:38:34.4760246Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_xr6bfzuv) 2022-11-23T01:38:34.4761626Z 2022-11-23T01:38:34.4762140Z 2022-11-23T01:38:34.4762909Z ##[endgroup] 2022-11-23T01:38:34.4764953Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_xr6bfzuv) 2022-11-23T01:38:34.4765873Z 2022-11-23T01:38:34.4774189Z Shard 1: nccl should be run in 2 2022-11-23T01:38:34.4775790Z Running distributed tests for the gloo backend with env init_method in shard 1 of 2 2022-11-23T01:38:34.4790405Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:38:34.478237] 2022-11-23T02:05:25.5230211Z 2022-11-23T02:05:25.5231464Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T02:05:25.5237533Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_ckaa7sv5) 2022-11-23T02:05:25.5250313Z 2022-11-23T02:05:25.5360477Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-11-23T02:05:25.5399775Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5402995Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5403517Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5404030Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5404591Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5405191Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5405771Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5406374Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5406943Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5407604Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5408277Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5408906Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5409542Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5410178Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5410763Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5411297Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5411861Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5412392Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5412916Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5413461Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5414043Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5414617Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5415064Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5415674Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5416203Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5416718Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5417256Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5417780Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5418288Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5418704Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5419188Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5419673Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5420157Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5420640Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5421166Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5421614Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5422129Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5422689Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5423207Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5423706Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5424210Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5424748Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5425233Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5425778Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5426299Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5426859Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5427407Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5427929Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5428431Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5428916Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5429453Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5429951Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5430462Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5430971Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5431489Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5431945Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5432439Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5432947Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5433429Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5433906Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5434406Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5434884Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5435296Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5435763Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5436294Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5436902Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5437394Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5437900Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5438325Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5438788Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5439292Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5439778Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5440255Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5440771Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5441196Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5441652Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5442120Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5442587Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5443082Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5443564Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5444141Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5444567Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5445101Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5445621Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5446156Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5446691Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5447237Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5447735Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5448294Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5448822Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5449365Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5449886Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5450442Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5450973Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5451495Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5452041Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5452597Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5453130Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5453669Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5454145Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5454621Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5455014Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5455492Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5455956Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5456474Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5456948Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5457406Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5457918Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5458409Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5458922Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5459408Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5459898Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5460425Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5460924Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5461397Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5461905Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5462435Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5462945Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5463470Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5463982Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5464406Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5464908Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5465394Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5465860Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5466370Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5466877Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5467440Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5468011Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5468560Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5469085Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5469583Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5470122Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5470658Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5471234Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5471744Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5472256Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5472794Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5473297Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5473783Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5474258Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5474708Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5475218Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5475758Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5476346Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5476857Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5477372Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5477926Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5478480Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5479207Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5479930Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5480649Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5481370Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5482079Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5482802Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5483519Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5484231Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5484932Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5485491Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5486024Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5486510Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5486992Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5487497Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5487989Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5488452Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5488977Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5489545Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5490107Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5490617Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5491097Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5491622Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5492102Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5492616Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5493123Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5493621Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5494148Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5494683Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5495195Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5495639Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5496170Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5496668Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5497171Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5497685Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5498230Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5498761Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5499253Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5499710Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5500209Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5500728Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5501231Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5501704Z test_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5502103Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5502551Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5503020Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5503480Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5503964Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5504430Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5504864Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5505314Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5505832Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5506309Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5506824Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5507306Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5507750Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5508188Z test_isend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5508650Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5509148Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5509653Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5510215Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5510783Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5511245Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5511760Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5512288Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5512801Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5513342Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5513838Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5514328Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5514824Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5515362Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5515839Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5516379Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5560796Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5561553Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5562086Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5562650Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5563200Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5563771Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5564485Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5564999Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5565516Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5566038Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5566562Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5567147Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5567780Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5568385Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5568965Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5569452Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5569897Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5570398Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5570866Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5571404Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5571866Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5572352Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5572720Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5573172Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5573620Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5574060Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5574547Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5575052Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5575474Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5575876Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5576356Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5576821Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5577272Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5577715Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5578179Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5578585Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5579057Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5579542Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5579954Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5580403Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5580848Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5581259Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5581764Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5582288Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5582808Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5583253Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5583727Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5584236Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5584699Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5585175Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5585708Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5586229Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5586728Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5587222Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5587666Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5588153Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5588586Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5589103Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5589651Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5590172Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5591430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5592182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5592751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5593448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5594026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5594296Z 2022-11-23T02:05:25.5594432Z Running tests... 2022-11-23T02:05:25.5594921Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5595639Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2669 2022-11-23T02:05:25.5596423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2670 2022-11-23T02:05:25.5596973Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5597746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5598289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5598975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5599539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5600108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5600849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5601331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5602015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5602603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5603155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5603920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5604751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5605368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5605962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5606598Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:05:25.5607603Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:05:25.5608375Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:05:25.5609360Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:05:25.5610094Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:05:25.5611022Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:05:25.5611790Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:05:25.5612760Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:05:25.5613541Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:05:25.5614148Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:05:25.5615033Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:05:25.5616036Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:05:25.5616750Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:05:25.5617360Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:05:25.5618256Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:05:25.5619244Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:05:25.5619746Z ok (4.438s) 2022-11-23T02:05:25.5619891Z 2022-11-23T02:05:25.5620186Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5620528Z Ran 1 test in 4.439s 2022-11-23T02:05:25.5620673Z 2022-11-23T02:05:25.5620758Z OK 2022-11-23T02:05:25.5620884Z 2022-11-23T02:05:25.5621003Z Generating XML reports... 2022-11-23T02:05:25.5621677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013837.xml 2022-11-23T02:05:25.5622398Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5623089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5623565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5624186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5624670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5624899Z 2022-11-23T02:05:25.5625000Z Running tests... 2022-11-23T02:05:25.5625439Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5625970Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s) 2022-11-23T02:05:25.5626349Z 2022-11-23T02:05:25.5626636Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5626969Z Ran 1 test in 0.003s 2022-11-23T02:05:25.5627133Z 2022-11-23T02:05:25.5627231Z OK (skipped=1) 2022-11-23T02:05:25.5627386Z 2022-11-23T02:05:25.5627489Z Generating XML reports... 2022-11-23T02:05:25.5628145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013845.xml 2022-11-23T02:05:25.5628864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5629538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5630011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5630636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5631143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5631373Z 2022-11-23T02:05:25.5631471Z Running tests... 2022-11-23T02:05:25.5631891Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5632531Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2914 2022-11-23T02:05:25.5633115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2915 2022-11-23T02:05:25.5633641Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5634349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5634827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5635462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5635937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5636483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5637164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5637629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5638258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5638749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5639236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5639957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5640819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5641515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5642183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5642665Z ok (3.834s) 2022-11-23T02:05:25.5642854Z 2022-11-23T02:05:25.5643235Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5643575Z Ran 1 test in 3.834s 2022-11-23T02:05:25.5643732Z 2022-11-23T02:05:25.5643814Z OK 2022-11-23T02:05:25.5643927Z 2022-11-23T02:05:25.5644046Z Generating XML reports... 2022-11-23T02:05:25.5644713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013848.xml 2022-11-23T02:05:25.5645498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5646164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5646645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5647275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5647768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5647999Z 2022-11-23T02:05:25.5648084Z Running tests... 2022-11-23T02:05:25.5648526Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5649851Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.508s) 2022-11-23T02:05:25.5650510Z 2022-11-23T02:05:25.5650805Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5651198Z Ran 1 test in 0.508s 2022-11-23T02:05:25.5651358Z 2022-11-23T02:05:25.5651457Z OK (skipped=1) 2022-11-23T02:05:25.5651609Z 2022-11-23T02:05:25.5651726Z Generating XML reports... 2022-11-23T02:05:25.5652403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013855.xml 2022-11-23T02:05:25.5653118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5653788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5654255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5654877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5655373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5655603Z 2022-11-23T02:05:25.5655702Z Running tests... 2022-11-23T02:05:25.5656146Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5656735Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3155 2022-11-23T02:05:25.5657328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3156 2022-11-23T02:05:25.5657857Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5658559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5659040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5659678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5660178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5660658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5661334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5661805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5662406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5662902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5663376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5664151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5664903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5665435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5665956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjx7_7ty2 2022-11-23T02:05:25.5666515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjx7_7ty2/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5667025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5667541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhkvd7lx 2022-11-23T02:05:25.5668087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhkvd7lx/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5668610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5669100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5669486Z ok (3.937s) 2022-11-23T02:05:25.5669627Z 2022-11-23T02:05:25.5669918Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5670230Z Ran 1 test in 3.938s 2022-11-23T02:05:25.5670380Z 2022-11-23T02:05:25.5670455Z OK 2022-11-23T02:05:25.5670574Z 2022-11-23T02:05:25.5670681Z Generating XML reports... 2022-11-23T02:05:25.5671336Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013858.xml 2022-11-23T02:05:25.5672035Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5672681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5673133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5673728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5674205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5674431Z 2022-11-23T02:05:25.5674523Z Running tests... 2022-11-23T02:05:25.5674942Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5675539Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3342 2022-11-23T02:05:25.5676223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3343 2022-11-23T02:05:25.5676741Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5677426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5677876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5678485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5678968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5679428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5680131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5680828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5681277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5681933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5682406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5682870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5683564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5684095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5684606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvnyeeng 2022-11-23T02:05:25.5685160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvnyeeng/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5685672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5686182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzprh_91 2022-11-23T02:05:25.5686728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzprh_91/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5687295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5687788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5688130Z ok (3.937s) 2022-11-23T02:05:25.5688265Z 2022-11-23T02:05:25.5688551Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5688864Z Ran 1 test in 3.937s 2022-11-23T02:05:25.5689014Z 2022-11-23T02:05:25.5689087Z OK 2022-11-23T02:05:25.5689205Z 2022-11-23T02:05:25.5689312Z Generating XML reports... 2022-11-23T02:05:25.5689963Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013905.xml 2022-11-23T02:05:25.5690662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5691316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5691768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5692366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5692844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5693066Z 2022-11-23T02:05:25.5693157Z Running tests... 2022-11-23T02:05:25.5693581Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5694182Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3529 2022-11-23T02:05:25.5694787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3530 2022-11-23T02:05:25.5695301Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5695994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5696440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5697040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5697507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5697967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5698635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5699136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5699749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5700217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5700682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5701385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5702125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5702660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5703138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5703656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5y8wcvl 2022-11-23T02:05:25.5704217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5y8wcvl/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5704753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidc2z41o 2022-11-23T02:05:25.5705350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidc2z41o/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5705881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5706375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5706869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5707364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5707851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5708328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5708812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5709294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5709782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5710259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5710739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5711216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5711549Z ok (7.444s) 2022-11-23T02:05:25.5711690Z 2022-11-23T02:05:25.5711979Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5712309Z Ran 1 test in 7.445s 2022-11-23T02:05:25.5712467Z 2022-11-23T02:05:25.5712542Z OK 2022-11-23T02:05:25.5712663Z 2022-11-23T02:05:25.5712777Z Generating XML reports... 2022-11-23T02:05:25.5713432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013912.xml 2022-11-23T02:05:25.5714140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5714794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5715254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5715870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5716415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5716640Z 2022-11-23T02:05:25.5716739Z Running tests... 2022-11-23T02:05:25.5717227Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5717845Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3722 2022-11-23T02:05:25.5718455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3723 2022-11-23T02:05:25.5719018Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5719719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5720173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5720790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5721269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5721745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5722453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5723198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5723661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5724269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5724746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5725207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5725914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5726456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5726938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5727450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85j7jqdl 2022-11-23T02:05:25.5728004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85j7jqdl/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5728553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9_zdcc8 2022-11-23T02:05:25.5729103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9_zdcc8/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5729669Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5730228Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5730750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5731242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5731736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5732233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5732581Z ok (4.439s) 2022-11-23T02:05:25.5732720Z 2022-11-23T02:05:25.5733013Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5733347Z Ran 1 test in 4.439s 2022-11-23T02:05:25.5733500Z 2022-11-23T02:05:25.5733575Z OK 2022-11-23T02:05:25.5733694Z 2022-11-23T02:05:25.5733797Z Generating XML reports... 2022-11-23T02:05:25.5734458Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013922.xml 2022-11-23T02:05:25.5735211Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5735877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5736340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5736961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5737447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5737673Z 2022-11-23T02:05:25.5737759Z Running tests... 2022-11-23T02:05:25.5738186Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5738805Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3913 2022-11-23T02:05:25.5739434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3914 2022-11-23T02:05:25.5739953Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5740689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5741153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5741762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5742229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5742693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5743365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5743820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5744432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5744912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5745378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5746078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5746824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5747365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5747848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5748368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgf0rtz5u 2022-11-23T02:05:25.5748926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgf0rtz5u/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5749475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pvndn5f 2022-11-23T02:05:25.5750028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pvndn5f/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5750548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5751043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5751535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5752026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5752514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5753046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5753529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5754007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5754348Z ok (4.427s) 2022-11-23T02:05:25.5754488Z 2022-11-23T02:05:25.5754777Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5755100Z Ran 1 test in 4.427s 2022-11-23T02:05:25.5755255Z 2022-11-23T02:05:25.5755334Z OK 2022-11-23T02:05:25.5755460Z 2022-11-23T02:05:25.5755569Z Generating XML reports... 2022-11-23T02:05:25.5756287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013930.xml 2022-11-23T02:05:25.5756983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5757640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5758102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5758765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5759249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5759477Z 2022-11-23T02:05:25.5759575Z Running tests... 2022-11-23T02:05:25.5760002Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5760646Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4104 2022-11-23T02:05:25.5761281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4105 2022-11-23T02:05:25.5761804Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5762496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5762950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5763560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5764041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5764510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5765164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5765619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5766227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5766701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5767160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5767860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5768602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5769140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5769617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5770127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy5q6ool 2022-11-23T02:05:25.5770755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy5q6ool/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5771307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpas1wrwtw 2022-11-23T02:05:25.5771860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpas1wrwtw/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5772382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5772874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5773213Z ok (4.738s) 2022-11-23T02:05:25.5773351Z 2022-11-23T02:05:25.5773642Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5773973Z Ran 1 test in 4.739s 2022-11-23T02:05:25.5774123Z 2022-11-23T02:05:25.5774200Z OK 2022-11-23T02:05:25.5774324Z 2022-11-23T02:05:25.5774439Z Generating XML reports... 2022-11-23T02:05:25.5775086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013937.xml 2022-11-23T02:05:25.5775790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5776487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5776948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5777560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5778039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5778267Z 2022-11-23T02:05:25.5778363Z Running tests... 2022-11-23T02:05:25.5778784Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5779419Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4295 2022-11-23T02:05:25.5780049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4296 2022-11-23T02:05:25.5780561Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5781252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5781709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5782317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5782795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5783260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5783930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5784377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5784986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5785464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5785930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5786628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5787362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5787900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5788419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5788941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkk14cen 2022-11-23T02:05:25.5789501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkk14cen/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5790058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0pfj3ff1 2022-11-23T02:05:25.5790610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0pfj3ff1/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5791145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5791639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5792130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5792610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5792959Z ok (7.342s) 2022-11-23T02:05:25.5793100Z 2022-11-23T02:05:25.5793394Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5793722Z Ran 1 test in 7.342s 2022-11-23T02:05:25.5793871Z 2022-11-23T02:05:25.5793946Z OK 2022-11-23T02:05:25.5794069Z 2022-11-23T02:05:25.5794257Z Generating XML reports... 2022-11-23T02:05:25.5794911Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013945.xml 2022-11-23T02:05:25.5795610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5796318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5796781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5797390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5797875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5798102Z 2022-11-23T02:05:25.5798193Z Running tests... 2022-11-23T02:05:25.5798625Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5799233Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4488 2022-11-23T02:05:25.5799849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4489 2022-11-23T02:05:25.5800366Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5801056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5801513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5802123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5802599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5803055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5803723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5804180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5804793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5805272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5805731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5806565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5807299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5807833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5808319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5808829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5omz797h 2022-11-23T02:05:25.5809389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5omz797h/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5809939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1u3gs3p 2022-11-23T02:05:25.5810492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1u3gs3p/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5811019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5811509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5812002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5812535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5812886Z ok (6.848s) 2022-11-23T02:05:25.5813025Z 2022-11-23T02:05:25.5813316Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5813647Z Ran 1 test in 6.849s 2022-11-23T02:05:25.5813801Z 2022-11-23T02:05:25.5813879Z OK 2022-11-23T02:05:25.5814001Z 2022-11-23T02:05:25.5814105Z Generating XML reports... 2022-11-23T02:05:25.5814761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013956.xml 2022-11-23T02:05:25.5815464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5816121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5816577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5817195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5817686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5817910Z 2022-11-23T02:05:25.5817995Z Running tests... 2022-11-23T02:05:25.5818424Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5819065Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4681 2022-11-23T02:05:25.5819707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4682 2022-11-23T02:05:25.5820234Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5820921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5821385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5821989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5822457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5822920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5823587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5824043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5824692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5825164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5825632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5826323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5827066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5827607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5828087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5828606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzj2snqy3 2022-11-23T02:05:25.5829161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzj2snqy3/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5829705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxs4axug5 2022-11-23T02:05:25.5830304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxs4axug5/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5830860Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5831421Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5831944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5832439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5832930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5833416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5833764Z ok (4.445s) 2022-11-23T02:05:25.5833906Z 2022-11-23T02:05:25.5834199Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5834523Z Ran 1 test in 4.446s 2022-11-23T02:05:25.5834678Z 2022-11-23T02:05:25.5834759Z OK 2022-11-23T02:05:25.5834883Z 2022-11-23T02:05:25.5834996Z Generating XML reports... 2022-11-23T02:05:25.5835646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014006.xml 2022-11-23T02:05:25.5836400Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5837055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5837509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5838125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5838600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5838824Z 2022-11-23T02:05:25.5838924Z Running tests... 2022-11-23T02:05:25.5839353Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5840637Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:05:25.5841326Z 2022-11-23T02:05:25.5841607Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5841985Z Ran 1 test in 0.509s 2022-11-23T02:05:25.5842140Z 2022-11-23T02:05:25.5842237Z OK (skipped=1) 2022-11-23T02:05:25.5842386Z 2022-11-23T02:05:25.5842489Z Generating XML reports... 2022-11-23T02:05:25.5843138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014013.xml 2022-11-23T02:05:25.5843839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5844499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5845003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5845705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5846273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5846545Z 2022-11-23T02:05:25.5846632Z Running tests... 2022-11-23T02:05:25.5847140Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5847813Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4930 2022-11-23T02:05:25.5848542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4931 2022-11-23T02:05:25.5849138Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5849913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5850449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5851142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5851652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5852197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5852937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5853466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5854165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5854717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5855260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5856001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5856821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5857442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5857998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5858422Z ok (3.840s) 2022-11-23T02:05:25.5858604Z 2022-11-23T02:05:25.5858926Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5859321Z Ran 1 test in 3.840s 2022-11-23T02:05:25.5859513Z 2022-11-23T02:05:25.5859582Z OK 2022-11-23T02:05:25.5859740Z 2022-11-23T02:05:25.5859890Z Generating XML reports... 2022-11-23T02:05:25.5860618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014017.xml 2022-11-23T02:05:25.5861397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5862130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5862722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5863411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5864019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5864237Z 2022-11-23T02:05:25.5864369Z Running tests... 2022-11-23T02:05:25.5864875Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5866251Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.508s) 2022-11-23T02:05:25.5866976Z 2022-11-23T02:05:25.5867292Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5867692Z Ran 1 test in 0.508s 2022-11-23T02:05:25.5867888Z 2022-11-23T02:05:25.5868017Z OK (skipped=1) 2022-11-23T02:05:25.5868199Z 2022-11-23T02:05:25.5868346Z Generating XML reports... 2022-11-23T02:05:25.5869134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014023.xml 2022-11-23T02:05:25.5869874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5870610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5871157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5871855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5872428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5872691Z 2022-11-23T02:05:25.5872824Z Running tests... 2022-11-23T02:05:25.5873331Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5873938Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5171 2022-11-23T02:05:25.5874589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5172 2022-11-23T02:05:25.5875188Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5875955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5876554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5877252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5877813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5878359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5879060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5879591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5880280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5880832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5881375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5882157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5883040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5883657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5884167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5884578Z ok (4.041s) 2022-11-23T02:05:25.5884759Z 2022-11-23T02:05:25.5885086Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5885490Z Ran 1 test in 4.042s 2022-11-23T02:05:25.5885681Z 2022-11-23T02:05:25.5885795Z OK 2022-11-23T02:05:25.5885955Z 2022-11-23T02:05:25.5886107Z Generating XML reports... 2022-11-23T02:05:25.5886785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014027.xml 2022-11-23T02:05:25.5887559Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5888292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5908902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5909769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5910264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5910495Z 2022-11-23T02:05:25.5910593Z Running tests... 2022-11-23T02:05:25.5911028Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5911529Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-11-23T02:05:25.5911824Z 2022-11-23T02:05:25.5912106Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5912435Z Ran 1 test in 0.002s 2022-11-23T02:05:25.5912588Z 2022-11-23T02:05:25.5912682Z OK (skipped=1) 2022-11-23T02:05:25.5912826Z 2022-11-23T02:05:25.5912937Z Generating XML reports... 2022-11-23T02:05:25.5913596Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014034.xml 2022-11-23T02:05:25.5914298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5914963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5915420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5916037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5916586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5916811Z 2022-11-23T02:05:25.5916907Z Running tests... 2022-11-23T02:05:25.5917347Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5917781Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5918333Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5412 2022-11-23T02:05:25.5918882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5413 2022-11-23T02:05:25.5919400Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5920089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5920555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5921168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5921723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5922184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5922860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5923319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5923929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5924403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5924873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5925573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5926326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5926865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5927384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8vq3bjq 2022-11-23T02:05:25.5928015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8vq3bjq/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5928529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5929034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpidy8ki 2022-11-23T02:05:25.5929589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpidy8ki/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5929968Z ok (3.751s) 2022-11-23T02:05:25.5930110Z 2022-11-23T02:05:25.5930404Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5930735Z Ran 1 test in 3.752s 2022-11-23T02:05:25.5930888Z 2022-11-23T02:05:25.5930967Z OK 2022-11-23T02:05:25.5931089Z 2022-11-23T02:05:25.5931193Z Generating XML reports... 2022-11-23T02:05:25.5931840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014037.xml 2022-11-23T02:05:25.5932537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5933193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5933650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5934265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5934752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5934975Z 2022-11-23T02:05:25.5935064Z Running tests... 2022-11-23T02:05:25.5935493Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5935950Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5936530Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5599 2022-11-23T02:05:25.5937081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5600 2022-11-23T02:05:25.5937601Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5938289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5938749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5939349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5939881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5940354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5941029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5941486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5942092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5942576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5943035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5943738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5944479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5945011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5945576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdj7244nh 2022-11-23T02:05:25.5946130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdj7244nh/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5946645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5947162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3_8tvt3 2022-11-23T02:05:25.5947699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3_8tvt3/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5948227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5948724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5949076Z ok (3.838s) 2022-11-23T02:05:25.5949213Z 2022-11-23T02:05:25.5949500Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5949824Z Ran 1 test in 3.838s 2022-11-23T02:05:25.5949982Z 2022-11-23T02:05:25.5950061Z OK 2022-11-23T02:05:25.5950174Z 2022-11-23T02:05:25.5950287Z Generating XML reports... 2022-11-23T02:05:25.5950937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014044.xml 2022-11-23T02:05:25.5951638Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5952300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5952755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5953368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5953850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5954074Z 2022-11-23T02:05:25.5954160Z Running tests... 2022-11-23T02:05:25.5954586Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5955056Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5955671Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5786 2022-11-23T02:05:25.5956282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5787 2022-11-23T02:05:25.5956802Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5957490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5957999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5958626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5959112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5959578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5960243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5960694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5961299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5961773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5962229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5962924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5963714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5964252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5964735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5965245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84xm1hy3 2022-11-23T02:05:25.5965799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84xm1hy3/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5966336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5cvr_b8 2022-11-23T02:05:25.5966885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5cvr_b8/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5967410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5967906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.5968258Z ok (3.840s) 2022-11-23T02:05:25.5968395Z 2022-11-23T02:05:25.5968686Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5969010Z Ran 1 test in 3.841s 2022-11-23T02:05:25.5969165Z 2022-11-23T02:05:25.5969235Z OK 2022-11-23T02:05:25.5969355Z 2022-11-23T02:05:25.5969463Z Generating XML reports... 2022-11-23T02:05:25.5970111Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014051.xml 2022-11-23T02:05:25.5970809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5971465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5971922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5972537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5973013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5973241Z 2022-11-23T02:05:25.5973335Z Running tests... 2022-11-23T02:05:25.5973759Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5974218Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:05:25.5974789Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5973 2022-11-23T02:05:25.5975384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5974 2022-11-23T02:05:25.5975902Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5976589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5977033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5977637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5978118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5978578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5979246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5979698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5980301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5980768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5981271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5981973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5982701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.5983233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.5983745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp945wnmj1 2022-11-23T02:05:25.5984298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp945wnmj1/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5984823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.5985333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhlrsif2 2022-11-23T02:05:25.5985891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhlrsif2/_remote_module_non_scriptable.py 2022-11-23T02:05:25.5986273Z ok (3.948s) 2022-11-23T02:05:25.5986413Z 2022-11-23T02:05:25.5986699Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5987023Z Ran 1 test in 3.949s 2022-11-23T02:05:25.5987176Z 2022-11-23T02:05:25.5987255Z OK 2022-11-23T02:05:25.5987372Z 2022-11-23T02:05:25.5987485Z Generating XML reports... 2022-11-23T02:05:25.5988124Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014058.xml 2022-11-23T02:05:25.5988822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.5989482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5989934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5990550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5991025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5991247Z 2022-11-23T02:05:25.5991342Z Running tests... 2022-11-23T02:05:25.5991760Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.5992298Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6160 2022-11-23T02:05:25.5992844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6161 2022-11-23T02:05:25.5993403Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.5994094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5994551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5995163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5995633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5996163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.5996830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.5997280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.5997894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.5998373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.5998841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.5999592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6000356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6000889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6001510Z STAGE:2022-11-23 01:41:07 6161:6161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6001994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6002600Z STAGE:2022-11-23 01:41:07 6160:6160 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6003222Z STAGE:2022-11-23 01:41:07 6161:6161 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6003869Z STAGE:2022-11-23 01:41:07 6161:6161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6004492Z STAGE:2022-11-23 01:41:07 6160:6160 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6005115Z STAGE:2022-11-23 01:41:07 6160:6160 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6005729Z STAGE:2022-11-23 01:41:07 6161:6161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6006329Z STAGE:2022-11-23 01:41:07 6160:6160 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6006950Z STAGE:2022-11-23 01:41:07 6161:6161 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6007587Z STAGE:2022-11-23 01:41:07 6161:6161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6008201Z STAGE:2022-11-23 01:41:07 6160:6160 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6008830Z STAGE:2022-11-23 01:41:07 6160:6160 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6009173Z ok (4.049s) 2022-11-23T02:05:25.6009307Z 2022-11-23T02:05:25.6009592Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6009913Z Ran 1 test in 4.050s 2022-11-23T02:05:25.6010061Z 2022-11-23T02:05:25.6010141Z OK 2022-11-23T02:05:25.6010260Z 2022-11-23T02:05:25.6010366Z Generating XML reports... 2022-11-23T02:05:25.6011014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014105.xml 2022-11-23T02:05:25.6011710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6012412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6012580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6012991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6013175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6013182Z 2022-11-23T02:05:25.6013274Z Running tests... 2022-11-23T02:05:25.6013554Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6013899Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6349 2022-11-23T02:05:25.6014114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6350 2022-11-23T02:05:25.6014387Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6014786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6014956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6015406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6015595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6015846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6016241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6016408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6016813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6017001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6017248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6017678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6018106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6018331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6018680Z STAGE:2022-11-23 01:41:14 6350:6350 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6018902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6019255Z STAGE:2022-11-23 01:41:14 6349:6349 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6019608Z STAGE:2022-11-23 01:41:14 6349:6349 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6019981Z STAGE:2022-11-23 01:41:14 6349:6349 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6020340Z STAGE:2022-11-23 01:41:14 6350:6350 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6020706Z STAGE:2022-11-23 01:41:14 6350:6350 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6021049Z STAGE:2022-11-23 01:41:14 6349:6349 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6021396Z STAGE:2022-11-23 01:41:14 6350:6350 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6021749Z STAGE:2022-11-23 01:41:14 6349:6349 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6022165Z STAGE:2022-11-23 01:41:14 6349:6349 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6022512Z STAGE:2022-11-23 01:41:14 6350:6350 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6022878Z STAGE:2022-11-23 01:41:14 6350:6350 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6023221Z STAGE:2022-11-23 01:41:14 6349:6349 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6023562Z STAGE:2022-11-23 01:41:14 6350:6350 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6023915Z STAGE:2022-11-23 01:41:14 6349:6349 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6024281Z STAGE:2022-11-23 01:41:14 6349:6349 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6024630Z STAGE:2022-11-23 01:41:14 6350:6350 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6024993Z STAGE:2022-11-23 01:41:14 6350:6350 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6025864Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6025966Z warnings.warn( 2022-11-23T02:05:25.6026776Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6026871Z warnings.warn( 2022-11-23T02:05:25.6026956Z ok (3.939s) 2022-11-23T02:05:25.6026964Z 2022-11-23T02:05:25.6027245Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6027339Z Ran 1 test in 3.940s 2022-11-23T02:05:25.6027346Z 2022-11-23T02:05:25.6027420Z OK 2022-11-23T02:05:25.6027427Z 2022-11-23T02:05:25.6027535Z Generating XML reports... 2022-11-23T02:05:25.6028021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014112.xml 2022-11-23T02:05:25.6028356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6028754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6028923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6029327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6029510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6029518Z 2022-11-23T02:05:25.6029614Z Running tests... 2022-11-23T02:05:25.6029897Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6030241Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6538 2022-11-23T02:05:25.6030455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6539 2022-11-23T02:05:25.6030730Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6031124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6031290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6031692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6031875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6032164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6032555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6032727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6033129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6033304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6033548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6033978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6034407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6034634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6034851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6035128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6035370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6035792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6036270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6036629Z STAGE:2022-11-23 01:41:21 6538:6538 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6036986Z STAGE:2022-11-23 01:41:21 6539:6539 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6037347Z STAGE:2022-11-23 01:41:21 6539:6539 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6037692Z STAGE:2022-11-23 01:41:21 6538:6538 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6038062Z STAGE:2022-11-23 01:41:21 6539:6539 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6038422Z STAGE:2022-11-23 01:41:21 6538:6538 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6038766Z STAGE:2022-11-23 01:41:21 6538:6538 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6039112Z STAGE:2022-11-23 01:41:21 6539:6539 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6039465Z STAGE:2022-11-23 01:41:21 6539:6539 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6039815Z STAGE:2022-11-23 01:41:21 6538:6538 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6040183Z STAGE:2022-11-23 01:41:21 6539:6539 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6040545Z STAGE:2022-11-23 01:41:21 6538:6538 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6040890Z STAGE:2022-11-23 01:41:21 6539:6539 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6041233Z STAGE:2022-11-23 01:41:21 6538:6538 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6041581Z STAGE:2022-11-23 01:41:21 6539:6539 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6041945Z STAGE:2022-11-23 01:41:21 6539:6539 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6042292Z STAGE:2022-11-23 01:41:21 6538:6538 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6042729Z STAGE:2022-11-23 01:41:21 6538:6538 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6043543Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6043642Z warnings.warn( 2022-11-23T02:05:25.6044446Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6044541Z warnings.warn( 2022-11-23T02:05:25.6044626Z ok (4.250s) 2022-11-23T02:05:25.6044634Z 2022-11-23T02:05:25.6044914Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6045015Z Ran 1 test in 4.250s 2022-11-23T02:05:25.6045023Z 2022-11-23T02:05:25.6045097Z OK 2022-11-23T02:05:25.6045104Z 2022-11-23T02:05:25.6045211Z Generating XML reports... 2022-11-23T02:05:25.6045693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014119.xml 2022-11-23T02:05:25.6046072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6046472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6046639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6047047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6047233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6047241Z 2022-11-23T02:05:25.6047332Z Running tests... 2022-11-23T02:05:25.6047616Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6047956Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6733 2022-11-23T02:05:25.6048173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6734 2022-11-23T02:05:25.6048441Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6048833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6048994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6049399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6049583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6049832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6050221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6050388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6050796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6050980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6051226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6051651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6052078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6052349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6052563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6052712Z skip: Skipped due to small world size. (3.737s) 2022-11-23T02:05:25.6052722Z 2022-11-23T02:05:25.6053005Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6053104Z Ran 1 test in 3.738s 2022-11-23T02:05:25.6053112Z 2022-11-23T02:05:25.6053204Z OK (skipped=1) 2022-11-23T02:05:25.6053211Z 2022-11-23T02:05:25.6053322Z Generating XML reports... 2022-11-23T02:05:25.6053813Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014126.xml 2022-11-23T02:05:25.6054145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6054537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6054709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6055102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6055330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6055339Z 2022-11-23T02:05:25.6055433Z Running tests... 2022-11-23T02:05:25.6055721Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6056062Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6916 2022-11-23T02:05:25.6056277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6917 2022-11-23T02:05:25.6056546Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6056939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6057107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6057515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6057699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6057944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6058373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6058764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6058933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6059342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6059525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6059769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6060192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6060417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6060778Z STAGE:2022-11-23 01:41:35 6917:6917 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6061001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6061352Z STAGE:2022-11-23 01:41:35 6916:6916 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6061754Z STAGE:2022-11-23 01:41:35 6916:6916 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6062123Z STAGE:2022-11-23 01:41:35 6916:6916 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6062479Z STAGE:2022-11-23 01:41:35 6917:6917 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6062841Z STAGE:2022-11-23 01:41:35 6917:6917 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6063186Z STAGE:2022-11-23 01:41:35 6916:6916 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6063531Z STAGE:2022-11-23 01:41:35 6917:6917 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6063882Z STAGE:2022-11-23 01:41:35 6917:6917 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6064246Z STAGE:2022-11-23 01:41:35 6917:6917 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6064598Z STAGE:2022-11-23 01:41:35 6916:6916 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6064957Z STAGE:2022-11-23 01:41:35 6916:6916 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6065351Z STAGE:2022-11-23 01:41:35 6916:6916 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6065701Z STAGE:2022-11-23 01:41:35 6917:6917 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6066057Z STAGE:2022-11-23 01:41:35 6916:6916 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6066424Z STAGE:2022-11-23 01:41:35 6916:6916 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6066774Z STAGE:2022-11-23 01:41:35 6917:6917 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6067140Z STAGE:2022-11-23 01:41:35 6917:6917 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6067956Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6068057Z warnings.warn( 2022-11-23T02:05:25.6068860Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6068961Z warnings.warn( 2022-11-23T02:05:25.6069046Z ok (4.043s) 2022-11-23T02:05:25.6069054Z 2022-11-23T02:05:25.6069338Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6069437Z Ran 1 test in 4.044s 2022-11-23T02:05:25.6069444Z 2022-11-23T02:05:25.6069522Z OK 2022-11-23T02:05:25.6069529Z 2022-11-23T02:05:25.6069638Z Generating XML reports... 2022-11-23T02:05:25.6070121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014133.xml 2022-11-23T02:05:25.6070458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6070853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6071019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6071427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6071611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6071619Z 2022-11-23T02:05:25.6071710Z Running tests... 2022-11-23T02:05:25.6071989Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6072381Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7105 2022-11-23T02:05:25.6072594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7106 2022-11-23T02:05:25.6072868Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6073264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6073431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6073838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6074023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6074269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6074664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6074829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6075274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6075459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6075702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6076186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6076612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6076836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6077190Z STAGE:2022-11-23 01:41:42 7106:7106 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6077419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6077771Z STAGE:2022-11-23 01:41:42 7105:7105 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6078131Z STAGE:2022-11-23 01:41:42 7106:7106 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6078502Z STAGE:2022-11-23 01:41:42 7106:7106 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6078851Z STAGE:2022-11-23 01:41:42 7105:7105 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6079215Z STAGE:2022-11-23 01:41:42 7105:7105 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6080025Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6080126Z warnings.warn( 2022-11-23T02:05:25.6080933Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6081029Z warnings.warn( 2022-11-23T02:05:25.6081116Z ok (3.837s) 2022-11-23T02:05:25.6081123Z 2022-11-23T02:05:25.6081404Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6081503Z Ran 1 test in 3.838s 2022-11-23T02:05:25.6081510Z 2022-11-23T02:05:25.6081586Z OK 2022-11-23T02:05:25.6081592Z 2022-11-23T02:05:25.6081758Z Generating XML reports... 2022-11-23T02:05:25.6082246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014140.xml 2022-11-23T02:05:25.6082579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6082975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6083144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6083552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6083735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6083743Z 2022-11-23T02:05:25.6083833Z Running tests... 2022-11-23T02:05:25.6084106Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6084433Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7294 2022-11-23T02:05:25.6084651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7295 2022-11-23T02:05:25.6084969Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6085366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6085531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6085936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6086119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6086366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6086759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6086922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6087326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6087510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6087751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6088177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6088602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6088829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6089188Z STAGE:2022-11-23 01:41:49 7295:7295 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6089416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6089769Z STAGE:2022-11-23 01:41:49 7294:7294 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6090130Z STAGE:2022-11-23 01:41:49 7294:7294 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6090482Z STAGE:2022-11-23 01:41:49 7295:7295 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6090852Z STAGE:2022-11-23 01:41:49 7294:7294 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6091207Z STAGE:2022-11-23 01:41:49 7295:7295 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6091552Z STAGE:2022-11-23 01:41:49 7295:7295 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6091941Z STAGE:2022-11-23 01:41:49 7294:7294 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6092296Z STAGE:2022-11-23 01:41:49 7294:7294 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6092663Z STAGE:2022-11-23 01:41:49 7294:7294 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6093015Z STAGE:2022-11-23 01:41:49 7295:7295 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6093375Z STAGE:2022-11-23 01:41:49 7295:7295 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6093461Z ok (3.827s) 2022-11-23T02:05:25.6093469Z 2022-11-23T02:05:25.6093751Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6093846Z Ran 1 test in 3.828s 2022-11-23T02:05:25.6093854Z 2022-11-23T02:05:25.6093930Z OK 2022-11-23T02:05:25.6093937Z 2022-11-23T02:05:25.6094049Z Generating XML reports... 2022-11-23T02:05:25.6094543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014147.xml 2022-11-23T02:05:25.6094879Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6095315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6095484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6095893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6096076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6096083Z 2022-11-23T02:05:25.6096175Z Running tests... 2022-11-23T02:05:25.6096455Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6096722Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s) 2022-11-23T02:05:25.6096732Z 2022-11-23T02:05:25.6097004Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6097092Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6097106Z 2022-11-23T02:05:25.6097189Z OK (skipped=1) 2022-11-23T02:05:25.6097195Z 2022-11-23T02:05:25.6097308Z Generating XML reports... 2022-11-23T02:05:25.6097797Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014154.xml 2022-11-23T02:05:25.6098128Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6098519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6098685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6099088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6099275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6099282Z 2022-11-23T02:05:25.6099372Z Running tests... 2022-11-23T02:05:25.6099646Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6099925Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-11-23T02:05:25.6099933Z 2022-11-23T02:05:25.6100203Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6100300Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6100307Z 2022-11-23T02:05:25.6100396Z OK (skipped=1) 2022-11-23T02:05:25.6100403Z 2022-11-23T02:05:25.6100509Z Generating XML reports... 2022-11-23T02:05:25.6100988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014157.xml 2022-11-23T02:05:25.6101315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6101785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6101953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6102357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6102540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6102547Z 2022-11-23T02:05:25.6102642Z Running tests... 2022-11-23T02:05:25.6102910Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6103240Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7599 2022-11-23T02:05:25.6103456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7600 2022-11-23T02:05:25.6103726Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6104125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6104290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6104738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6104925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6105173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6105566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6105735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6106135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6106321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6106566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6107001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6107427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6107655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6107880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6108118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6108360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6108785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6109207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6109564Z STAGE:2022-11-23 01:42:02 7599:7599 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6109913Z STAGE:2022-11-23 01:42:02 7600:7600 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6110264Z STAGE:2022-11-23 01:42:02 7600:7600 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6110633Z STAGE:2022-11-23 01:42:02 7600:7600 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6110983Z STAGE:2022-11-23 01:42:02 7599:7599 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6111398Z STAGE:2022-11-23 01:42:02 7599:7599 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6111740Z STAGE:2022-11-23 01:42:02 7600:7600 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6112096Z STAGE:2022-11-23 01:42:02 7599:7599 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6112449Z STAGE:2022-11-23 01:42:02 7599:7599 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6112813Z STAGE:2022-11-23 01:42:02 7599:7599 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6113166Z STAGE:2022-11-23 01:42:02 7600:7600 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6113526Z STAGE:2022-11-23 01:42:02 7600:7600 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6113612Z ok (3.837s) 2022-11-23T02:05:25.6113621Z 2022-11-23T02:05:25.6113906Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6114006Z Ran 1 test in 3.838s 2022-11-23T02:05:25.6114013Z 2022-11-23T02:05:25.6114088Z OK 2022-11-23T02:05:25.6114093Z 2022-11-23T02:05:25.6114204Z Generating XML reports... 2022-11-23T02:05:25.6114750Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014200.xml 2022-11-23T02:05:25.6115089Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6115482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6115652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6116155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6116342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6116353Z 2022-11-23T02:05:25.6116447Z Running tests... 2022-11-23T02:05:25.6116721Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6117045Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7794 2022-11-23T02:05:25.6117260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7795 2022-11-23T02:05:25.6117534Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6117926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6118094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6118497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6118685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6118927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6119323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6119493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6119898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6120077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6120320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6120751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6121237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6121463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6121691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6121844Z skip: Skipped due to small world size. (3.836s) 2022-11-23T02:05:25.6121852Z 2022-11-23T02:05:25.6122132Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6122230Z Ran 1 test in 3.837s 2022-11-23T02:05:25.6122237Z 2022-11-23T02:05:25.6122329Z OK (skipped=1) 2022-11-23T02:05:25.6122335Z 2022-11-23T02:05:25.6122447Z Generating XML reports... 2022-11-23T02:05:25.6122928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014207.xml 2022-11-23T02:05:25.6123260Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6123655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6123824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6124275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6124462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6124470Z 2022-11-23T02:05:25.6124562Z Running tests... 2022-11-23T02:05:25.6124844Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6125147Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2022-11-23T02:05:25.6125155Z 2022-11-23T02:05:25.6125433Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6125532Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6125539Z 2022-11-23T02:05:25.6125631Z OK (skipped=1) 2022-11-23T02:05:25.6125637Z 2022-11-23T02:05:25.6125750Z Generating XML reports... 2022-11-23T02:05:25.6126234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014214.xml 2022-11-23T02:05:25.6126569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6126962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6127128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6127534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6127716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6127723Z 2022-11-23T02:05:25.6127814Z Running tests... 2022-11-23T02:05:25.6128095Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6128401Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2022-11-23T02:05:25.6128409Z 2022-11-23T02:05:25.6128686Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6128777Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6128790Z 2022-11-23T02:05:25.6128872Z OK (skipped=1) 2022-11-23T02:05:25.6128878Z 2022-11-23T02:05:25.6128987Z Generating XML reports... 2022-11-23T02:05:25.6129469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014216.xml 2022-11-23T02:05:25.6129802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6130196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6130414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6130822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6131007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6131014Z 2022-11-23T02:05:25.6131108Z Running tests... 2022-11-23T02:05:25.6131384Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6131673Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-11-23T02:05:25.6131680Z 2022-11-23T02:05:25.6131948Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6132045Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6132052Z 2022-11-23T02:05:25.6132143Z OK (skipped=1) 2022-11-23T02:05:25.6132149Z 2022-11-23T02:05:25.6132260Z Generating XML reports... 2022-11-23T02:05:25.6132748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014219.xml 2022-11-23T02:05:25.6133078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6133516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6133685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6134093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6134274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6134281Z 2022-11-23T02:05:25.6134372Z Running tests... 2022-11-23T02:05:25.6134641Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6134947Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-11-23T02:05:25.6134958Z 2022-11-23T02:05:25.6135229Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6135326Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6135332Z 2022-11-23T02:05:25.6135422Z OK (skipped=1) 2022-11-23T02:05:25.6135429Z 2022-11-23T02:05:25.6135541Z Generating XML reports... 2022-11-23T02:05:25.6136022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014222.xml 2022-11-23T02:05:25.6136351Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6136742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6136909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6137312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6137499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6137506Z 2022-11-23T02:05:25.6137598Z Running tests... 2022-11-23T02:05:25.6137871Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6138214Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8209 2022-11-23T02:05:25.6138430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8210 2022-11-23T02:05:25.6138704Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6139098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6139266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6139717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6139898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6140144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6140580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6140964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6141131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6141535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6141720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6141966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6142393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6142664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6142891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6142976Z ok (3.747s) 2022-11-23T02:05:25.6142984Z 2022-11-23T02:05:25.6143262Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6143359Z Ran 1 test in 3.748s 2022-11-23T02:05:25.6143367Z 2022-11-23T02:05:25.6143443Z OK 2022-11-23T02:05:25.6143449Z 2022-11-23T02:05:25.6143557Z Generating XML reports... 2022-11-23T02:05:25.6144043Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014225.xml 2022-11-23T02:05:25.6144381Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6144775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6144943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6145354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6145538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6145545Z 2022-11-23T02:05:25.6145639Z Running tests... 2022-11-23T02:05:25.6145915Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6146247Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8392 2022-11-23T02:05:25.6146453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8393 2022-11-23T02:05:25.6146726Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6147119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6147290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6147691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6147877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6148126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6148518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6148843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6149304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6149487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6149737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6150165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6150589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6150814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6151040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6151278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6151522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6151943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6152409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6152650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:05:25.6152885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:05:25.6153316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:05:25.6153737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:05:25.6153973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T02:05:25.6154204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T02:05:25.6154627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:05:25.6155050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:05:25.6155137Z ok (4.039s) 2022-11-23T02:05:25.6155145Z 2022-11-23T02:05:25.6155425Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6155521Z Ran 1 test in 4.040s 2022-11-23T02:05:25.6155528Z 2022-11-23T02:05:25.6155602Z OK 2022-11-23T02:05:25.6155608Z 2022-11-23T02:05:25.6155714Z Generating XML reports... 2022-11-23T02:05:25.6156251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014232.xml 2022-11-23T02:05:25.6156588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6156988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6157158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6157562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6157747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6157754Z 2022-11-23T02:05:25.6157846Z Running tests... 2022-11-23T02:05:25.6158126Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6158388Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s) 2022-11-23T02:05:25.6158446Z 2022-11-23T02:05:25.6158719Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6158816Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6158823Z 2022-11-23T02:05:25.6158913Z OK (skipped=1) 2022-11-23T02:05:25.6158919Z 2022-11-23T02:05:25.6159027Z Generating XML reports... 2022-11-23T02:05:25.6159512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014239.xml 2022-11-23T02:05:25.6159835Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6160234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6160403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6160808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6160994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6161001Z 2022-11-23T02:05:25.6161095Z Running tests... 2022-11-23T02:05:25.6161373Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6161774Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8657 2022-11-23T02:05:25.6161989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8658 2022-11-23T02:05:25.6162263Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6162660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6162826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6163230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6163414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6163657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6164051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6164216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6164622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6164806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6165047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6165474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6165902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6166131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6166352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6166588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6166821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6167251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6167672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6168075Z STAGE:2022-11-23 01:42:44 8657:8657 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6168425Z STAGE:2022-11-23 01:42:44 8658:8658 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6168780Z STAGE:2022-11-23 01:42:44 8657:8657 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6169156Z STAGE:2022-11-23 01:42:44 8657:8657 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6169512Z STAGE:2022-11-23 01:42:44 8658:8658 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6169876Z STAGE:2022-11-23 01:42:44 8658:8658 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6170219Z STAGE:2022-11-23 01:42:44 8657:8657 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6170560Z STAGE:2022-11-23 01:42:44 8658:8658 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6170920Z STAGE:2022-11-23 01:42:44 8658:8658 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6171285Z STAGE:2022-11-23 01:42:44 8658:8658 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6171638Z STAGE:2022-11-23 01:42:44 8657:8657 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6172040Z STAGE:2022-11-23 01:42:44 8657:8657 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6172858Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6172956Z warnings.warn( 2022-11-23T02:05:25.6173764Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6173863Z warnings.warn( 2022-11-23T02:05:25.6173947Z ok (3.939s) 2022-11-23T02:05:25.6173955Z 2022-11-23T02:05:25.6174238Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6174336Z Ran 1 test in 3.939s 2022-11-23T02:05:25.6174344Z 2022-11-23T02:05:25.6174418Z OK 2022-11-23T02:05:25.6174424Z 2022-11-23T02:05:25.6174526Z Generating XML reports... 2022-11-23T02:05:25.6175012Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014242.xml 2022-11-23T02:05:25.6175344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6175739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6175913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6176321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6176505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6176513Z 2022-11-23T02:05:25.6176607Z Running tests... 2022-11-23T02:05:25.6176887Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6177237Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8852 2022-11-23T02:05:25.6177453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8853 2022-11-23T02:05:25.6177724Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6178116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6178329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6178733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6178919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6179163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6179553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6179717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6180121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6180308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6180556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6180985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6181444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6181673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6181892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6182130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6182370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6182800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6183228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6183583Z STAGE:2022-11-23 01:42:51 8852:8852 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6183941Z STAGE:2022-11-23 01:42:51 8853:8853 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6184299Z STAGE:2022-11-23 01:42:51 8852:8852 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6184646Z STAGE:2022-11-23 01:42:51 8853:8853 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6185015Z STAGE:2022-11-23 01:42:51 8852:8852 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6185374Z STAGE:2022-11-23 01:42:51 8853:8853 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6185719Z STAGE:2022-11-23 01:42:51 8853:8853 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6186065Z STAGE:2022-11-23 01:42:51 8852:8852 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6186420Z STAGE:2022-11-23 01:42:51 8852:8852 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6186790Z STAGE:2022-11-23 01:42:51 8852:8852 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6187139Z STAGE:2022-11-23 01:42:51 8853:8853 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6187495Z STAGE:2022-11-23 01:42:51 8853:8853 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6188309Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6188472Z warnings.warn( 2022-11-23T02:05:25.6189289Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6189386Z warnings.warn( 2022-11-23T02:05:25.6189469Z ok (3.847s) 2022-11-23T02:05:25.6189476Z 2022-11-23T02:05:25.6189761Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6189859Z Ran 1 test in 3.847s 2022-11-23T02:05:25.6189866Z 2022-11-23T02:05:25.6189937Z OK 2022-11-23T02:05:25.6189952Z 2022-11-23T02:05:25.6190055Z Generating XML reports... 2022-11-23T02:05:25.6190544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014249.xml 2022-11-23T02:05:25.6190877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6191280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6191451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6191904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6192091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6192098Z 2022-11-23T02:05:25.6192191Z Running tests... 2022-11-23T02:05:25.6192477Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6192835Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9047 2022-11-23T02:05:25.6193051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9048 2022-11-23T02:05:25.6193330Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6193721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6193892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6194296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6194479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6194722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6195151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6195541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6195710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6196175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6196358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6196589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6197015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6197245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6197465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6197706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6198000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6198430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6198857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6199214Z STAGE:2022-11-23 01:42:58 9047:9047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6199566Z STAGE:2022-11-23 01:42:58 9048:9048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6199928Z STAGE:2022-11-23 01:42:58 9048:9048 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6200279Z STAGE:2022-11-23 01:42:58 9047:9047 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6200647Z STAGE:2022-11-23 01:42:58 9048:9048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6201012Z STAGE:2022-11-23 01:42:58 9047:9047 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6201355Z STAGE:2022-11-23 01:42:58 9048:9048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6201753Z STAGE:2022-11-23 01:42:58 9047:9047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6202110Z STAGE:2022-11-23 01:42:58 9047:9047 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6202477Z STAGE:2022-11-23 01:42:58 9047:9047 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6202827Z STAGE:2022-11-23 01:42:58 9048:9048 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6203186Z STAGE:2022-11-23 01:42:58 9048:9048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6203999Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6204101Z warnings.warn( 2022-11-23T02:05:25.6204913Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6205010Z warnings.warn( 2022-11-23T02:05:25.6205096Z ok (3.838s) 2022-11-23T02:05:25.6205103Z 2022-11-23T02:05:25.6205389Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6205480Z Ran 1 test in 3.839s 2022-11-23T02:05:25.6205487Z 2022-11-23T02:05:25.6205562Z OK 2022-11-23T02:05:25.6205568Z 2022-11-23T02:05:25.6205681Z Generating XML reports... 2022-11-23T02:05:25.6206171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014256.xml 2022-11-23T02:05:25.6206503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6206900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6207069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6207475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6207659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6207667Z 2022-11-23T02:05:25.6207758Z Running tests... 2022-11-23T02:05:25.6208037Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6208382Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9242 2022-11-23T02:05:25.6208646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9243 2022-11-23T02:05:25.6208918Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6209319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6209485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6209888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6210073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6210317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6210746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6211134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6211341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6211745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6211931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6212174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6212596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6212825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6213048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6213293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6213538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6213970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6214401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6214762Z STAGE:2022-11-23 01:43:05 9242:9242 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6215117Z STAGE:2022-11-23 01:43:05 9243:9243 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6215477Z STAGE:2022-11-23 01:43:05 9243:9243 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6215852Z STAGE:2022-11-23 01:43:05 9243:9243 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6216212Z STAGE:2022-11-23 01:43:05 9242:9242 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6216580Z STAGE:2022-11-23 01:43:05 9242:9242 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6216928Z STAGE:2022-11-23 01:43:05 9242:9242 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6217277Z STAGE:2022-11-23 01:43:05 9243:9243 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6217636Z STAGE:2022-11-23 01:43:05 9243:9243 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6218003Z STAGE:2022-11-23 01:43:05 9243:9243 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6218355Z STAGE:2022-11-23 01:43:05 9242:9242 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6218768Z STAGE:2022-11-23 01:43:05 9242:9242 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6219586Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6219690Z warnings.warn( 2022-11-23T02:05:25.6220505Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6220607Z warnings.warn( 2022-11-23T02:05:25.6220685Z ok (3.855s) 2022-11-23T02:05:25.6220704Z 2022-11-23T02:05:25.6220981Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6221081Z Ran 1 test in 3.856s 2022-11-23T02:05:25.6221089Z 2022-11-23T02:05:25.6221165Z OK 2022-11-23T02:05:25.6221171Z 2022-11-23T02:05:25.6221283Z Generating XML reports... 2022-11-23T02:05:25.6221909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014303.xml 2022-11-23T02:05:25.6222248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6222645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6222817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6223225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6223412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6223423Z 2022-11-23T02:05:25.6223522Z Running tests... 2022-11-23T02:05:25.6223808Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6224159Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9437 2022-11-23T02:05:25.6224381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9438 2022-11-23T02:05:25.6224661Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6225061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6225234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6225641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6225834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6226081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6226477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6226638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6227047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6227235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6227482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6227916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6228344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6228620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6228842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6228997Z skip: Skipped due to small world size. (3.737s) 2022-11-23T02:05:25.6229004Z 2022-11-23T02:05:25.6229290Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6229392Z Ran 1 test in 3.738s 2022-11-23T02:05:25.6229399Z 2022-11-23T02:05:25.6229491Z OK (skipped=1) 2022-11-23T02:05:25.6229498Z 2022-11-23T02:05:25.6229609Z Generating XML reports... 2022-11-23T02:05:25.6230097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014310.xml 2022-11-23T02:05:25.6230431Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6230829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6231001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6231450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6231638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6231646Z 2022-11-23T02:05:25.6231742Z Running tests... 2022-11-23T02:05:25.6232029Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6232375Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9620 2022-11-23T02:05:25.6232594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9621 2022-11-23T02:05:25.6232860Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6233257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6233424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6233833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6234021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6234266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6234698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6235090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6235259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6235662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6235846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6236148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6236575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6236804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6237031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6237183Z skip: Skipped due to small world size. (3.639s) 2022-11-23T02:05:25.6237191Z 2022-11-23T02:05:25.6237470Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6237622Z Ran 1 test in 3.640s 2022-11-23T02:05:25.6237630Z 2022-11-23T02:05:25.6237721Z OK (skipped=1) 2022-11-23T02:05:25.6237728Z 2022-11-23T02:05:25.6237839Z Generating XML reports... 2022-11-23T02:05:25.6238335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014316.xml 2022-11-23T02:05:25.6238677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6239076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6239238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6239641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6239827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6239838Z 2022-11-23T02:05:25.6239933Z Running tests... 2022-11-23T02:05:25.6240217Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6240572Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9803 2022-11-23T02:05:25.6240843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9804 2022-11-23T02:05:25.6241118Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6241517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6241684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6242087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6242274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6242519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6242909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6243085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6243492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6243675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6243918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6244351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6244784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6245016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6245236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6245391Z skip: Skipped due to small world size. (3.734s) 2022-11-23T02:05:25.6245398Z 2022-11-23T02:05:25.6245672Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6245770Z Ran 1 test in 3.735s 2022-11-23T02:05:25.6245778Z 2022-11-23T02:05:25.6245869Z OK (skipped=1) 2022-11-23T02:05:25.6245876Z 2022-11-23T02:05:25.6245985Z Generating XML reports... 2022-11-23T02:05:25.6246471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014323.xml 2022-11-23T02:05:25.6246803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6247245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6247413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6247822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6248007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6248014Z 2022-11-23T02:05:25.6248107Z Running tests... 2022-11-23T02:05:25.6248388Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6248732Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9986 2022-11-23T02:05:25.6248949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9987 2022-11-23T02:05:25.6249223Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6249614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6249781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6250228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6250413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6250810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6251208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6251377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6251775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6251961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6252206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6252668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6253134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6253403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6253671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6253865Z skip: Skipped due to small world size. (3.741s) 2022-11-23T02:05:25.6253871Z 2022-11-23T02:05:25.6254196Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6254341Z Ran 1 test in 3.742s 2022-11-23T02:05:25.6254349Z 2022-11-23T02:05:25.6254483Z OK (skipped=1) 2022-11-23T02:05:25.6254490Z 2022-11-23T02:05:25.6254649Z Generating XML reports... 2022-11-23T02:05:25.6255181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014330.xml 2022-11-23T02:05:25.6255551Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6255982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6256194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6256638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6256860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6256914Z 2022-11-23T02:05:25.6257047Z Running tests... 2022-11-23T02:05:25.6257369Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6257747Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10169 2022-11-23T02:05:25.6258006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10170 2022-11-23T02:05:25.6258318Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6258704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6258914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6259365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6259587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6259870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6260341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6260834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6261043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6261490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6261714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6261995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6262460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6262732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6263140Z STAGE:2022-11-23 01:43:39 10170:10170 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6263406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6263810Z STAGE:2022-11-23 01:43:39 10169:10169 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6264216Z STAGE:2022-11-23 01:43:39 10170:10170 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6264634Z STAGE:2022-11-23 01:43:39 10170:10170 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6265037Z STAGE:2022-11-23 01:43:39 10169:10169 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6265458Z STAGE:2022-11-23 01:43:39 10169:10169 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6265853Z STAGE:2022-11-23 01:43:39 10170:10170 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6266236Z STAGE:2022-11-23 01:43:39 10169:10169 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6266636Z STAGE:2022-11-23 01:43:39 10169:10169 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6267048Z STAGE:2022-11-23 01:43:39 10169:10169 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6267400Z STAGE:2022-11-23 01:43:39 10170:10170 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6267806Z STAGE:2022-11-23 01:43:39 10170:10170 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6268665Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6268861Z warnings.warn( 2022-11-23T02:05:25.6269719Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6269854Z warnings.warn( 2022-11-23T02:05:25.6269980Z ok (3.849s) 2022-11-23T02:05:25.6269988Z 2022-11-23T02:05:25.6270311Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6270452Z Ran 1 test in 3.850s 2022-11-23T02:05:25.6270459Z 2022-11-23T02:05:25.6270580Z OK 2022-11-23T02:05:25.6270587Z 2022-11-23T02:05:25.6270742Z Generating XML reports... 2022-11-23T02:05:25.6271274Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014337.xml 2022-11-23T02:05:25.6271806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6272453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6272810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6273279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6273503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6273510Z 2022-11-23T02:05:25.6273646Z Running tests... 2022-11-23T02:05:25.6273971Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6274385Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10358 2022-11-23T02:05:25.6274645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10359 2022-11-23T02:05:25.6274960Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6275399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6275609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6276097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6276274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6276559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6276997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6277215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6277663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6277895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6278181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6278654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6279121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6279388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6279648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6280556Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6288057Z warnings.warn( 2022-11-23T02:05:25.6289007Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6289107Z warnings.warn( 2022-11-23T02:05:25.6289193Z ok (3.938s) 2022-11-23T02:05:25.6289202Z 2022-11-23T02:05:25.6289493Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6289590Z Ran 1 test in 3.939s 2022-11-23T02:05:25.6289603Z 2022-11-23T02:05:25.6289677Z OK 2022-11-23T02:05:25.6289683Z 2022-11-23T02:05:25.6289793Z Generating XML reports... 2022-11-23T02:05:25.6290291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014344.xml 2022-11-23T02:05:25.6290751Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6291158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6291322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6291728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6291914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6291922Z 2022-11-23T02:05:25.6292013Z Running tests... 2022-11-23T02:05:25.6292295Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6292637Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10541 2022-11-23T02:05:25.6292860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10542 2022-11-23T02:05:25.6293141Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6293539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6293711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6294108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6294292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6294539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6294933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6295101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6295510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6295693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6295939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6296373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6296805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6297097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6297468Z STAGE:2022-11-23 01:43:53 10542:10542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6297691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6298057Z STAGE:2022-11-23 01:43:53 10541:10541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6298419Z STAGE:2022-11-23 01:43:53 10541:10541 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6298795Z STAGE:2022-11-23 01:43:53 10541:10541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6299155Z STAGE:2022-11-23 01:43:53 10542:10542 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6299525Z STAGE:2022-11-23 01:43:53 10542:10542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6299876Z STAGE:2022-11-23 01:43:53 10541:10541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6300220Z STAGE:2022-11-23 01:43:53 10542:10542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6300577Z STAGE:2022-11-23 01:43:53 10541:10541 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6301021Z STAGE:2022-11-23 01:43:53 10541:10541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6301380Z STAGE:2022-11-23 01:43:53 10542:10542 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6301745Z STAGE:2022-11-23 01:43:53 10542:10542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6302557Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6302653Z warnings.warn( 2022-11-23T02:05:25.6303474Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6303573Z warnings.warn( 2022-11-23T02:05:25.6303662Z ok (3.849s) 2022-11-23T02:05:25.6303670Z 2022-11-23T02:05:25.6303955Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6304053Z Ran 1 test in 3.850s 2022-11-23T02:05:25.6304061Z 2022-11-23T02:05:25.6304136Z OK 2022-11-23T02:05:25.6304142Z 2022-11-23T02:05:25.6304252Z Generating XML reports... 2022-11-23T02:05:25.6304744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014351.xml 2022-11-23T02:05:25.6305075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6305473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6305643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6306055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6306241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6306250Z 2022-11-23T02:05:25.6306342Z Running tests... 2022-11-23T02:05:25.6306624Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6306974Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10730 2022-11-23T02:05:25.6307193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10731 2022-11-23T02:05:25.6307514Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6307911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6308085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6308492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6308675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6308913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6309345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6309733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6309902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6310307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6310531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6310773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6311202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6311428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6311792Z STAGE:2022-11-23 01:44:00 10731:10731 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6312014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6312370Z STAGE:2022-11-23 01:44:00 10730:10730 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6312732Z STAGE:2022-11-23 01:44:00 10731:10731 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6313088Z STAGE:2022-11-23 01:44:00 10730:10730 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6313460Z STAGE:2022-11-23 01:44:00 10731:10731 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6313833Z STAGE:2022-11-23 01:44:00 10730:10730 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6314182Z STAGE:2022-11-23 01:44:00 10731:10731 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6314525Z STAGE:2022-11-23 01:44:00 10730:10730 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6314880Z STAGE:2022-11-23 01:44:00 10730:10730 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6315252Z STAGE:2022-11-23 01:44:00 10730:10730 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6315605Z STAGE:2022-11-23 01:44:00 10731:10731 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6315977Z STAGE:2022-11-23 01:44:00 10731:10731 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6316875Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6316973Z warnings.warn( 2022-11-23T02:05:25.6317784Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6317936Z warnings.warn( 2022-11-23T02:05:25.6318014Z ok (3.940s) 2022-11-23T02:05:25.6318028Z 2022-11-23T02:05:25.6318307Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6318403Z Ran 1 test in 3.940s 2022-11-23T02:05:25.6318413Z 2022-11-23T02:05:25.6318487Z OK 2022-11-23T02:05:25.6318494Z 2022-11-23T02:05:25.6318604Z Generating XML reports... 2022-11-23T02:05:25.6319095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014358.xml 2022-11-23T02:05:25.6319429Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6319822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6319989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6320401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6320586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6320593Z 2022-11-23T02:05:25.6320686Z Running tests... 2022-11-23T02:05:25.6321016Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6321358Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10919 2022-11-23T02:05:25.6321572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10920 2022-11-23T02:05:25.6321840Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6322231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6322400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6322807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6322989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6323235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6323628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6323785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6324188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6324372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6324616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6325050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6325475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6325703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6325918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6326281Z STAGE:2022-11-23 01:44:07 10919:10919 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6326632Z STAGE:2022-11-23 01:44:07 10920:10920 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6327240Z STAGE:2022-11-23 01:44:07 10920:10920 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:44:07 10919:10919 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6327292Z 2022-11-23T02:05:25.6327673Z STAGE:2022-11-23 01:44:07 10919:10919 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6328044Z STAGE:2022-11-23 01:44:07 10920:10920 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6328395Z STAGE:2022-11-23 01:44:07 10920:10920 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6328739Z STAGE:2022-11-23 01:44:07 10919:10919 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6329097Z STAGE:2022-11-23 01:44:07 10919:10919 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6329465Z STAGE:2022-11-23 01:44:07 10919:10919 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6329823Z STAGE:2022-11-23 01:44:07 10920:10920 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6330199Z STAGE:2022-11-23 01:44:07 10920:10920 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6331055Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6331155Z warnings.warn( 2022-11-23T02:05:25.6331968Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:05:25.6332063Z warnings.warn( 2022-11-23T02:05:25.6332146Z ok (3.741s) 2022-11-23T02:05:25.6332153Z 2022-11-23T02:05:25.6332440Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6332541Z Ran 1 test in 3.741s 2022-11-23T02:05:25.6332549Z 2022-11-23T02:05:25.6332622Z OK 2022-11-23T02:05:25.6332629Z 2022-11-23T02:05:25.6332739Z Generating XML reports... 2022-11-23T02:05:25.6333220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014404.xml 2022-11-23T02:05:25.6333554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6333953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6334123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6334526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6334711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6334719Z 2022-11-23T02:05:25.6334812Z Running tests... 2022-11-23T02:05:25.6335092Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6335443Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11108 2022-11-23T02:05:25.6335660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11109 2022-11-23T02:05:25.6335930Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6336320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6336486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6336884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6337068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6337365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6337758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6337925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6338326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6338507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6338753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6339181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6339608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6339829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6340047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6340132Z ok (3.848s) 2022-11-23T02:05:25.6340183Z 2022-11-23T02:05:25.6340466Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6340562Z Ran 1 test in 3.848s 2022-11-23T02:05:25.6340570Z 2022-11-23T02:05:25.6340643Z OK 2022-11-23T02:05:25.6340650Z 2022-11-23T02:05:25.6340758Z Generating XML reports... 2022-11-23T02:05:25.6341247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014411.xml 2022-11-23T02:05:25.6341576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6341969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6342137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6342539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6342724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6342731Z 2022-11-23T02:05:25.6342823Z Running tests... 2022-11-23T02:05:25.6343100Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6343433Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11291 2022-11-23T02:05:25.6343649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11292 2022-11-23T02:05:25.6343921Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6344318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6344485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6344891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6345072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6345306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6345735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6346127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6346291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6346759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6346938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6347176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6350229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6350456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6350685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6350928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6351168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6351602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6352027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6352479Z STAGE:2022-11-23 01:44:21 11291:11291 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6352840Z STAGE:2022-11-23 01:44:21 11292:11292 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6353203Z STAGE:2022-11-23 01:44:21 11292:11292 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6353553Z STAGE:2022-11-23 01:44:21 11291:11291 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6353925Z STAGE:2022-11-23 01:44:21 11292:11292 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6354300Z STAGE:2022-11-23 01:44:21 11291:11291 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6354650Z STAGE:2022-11-23 01:44:21 11291:11291 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6355003Z STAGE:2022-11-23 01:44:21 11292:11292 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6355364Z STAGE:2022-11-23 01:44:21 11292:11292 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6355736Z STAGE:2022-11-23 01:44:21 11292:11292 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6356146Z STAGE:2022-11-23 01:44:21 11291:11291 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6356504Z STAGE:2022-11-23 01:44:21 11291:11291 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6356592Z ok (4.138s) 2022-11-23T02:05:25.6356600Z 2022-11-23T02:05:25.6356883Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6356982Z Ran 1 test in 4.139s 2022-11-23T02:05:25.6356989Z 2022-11-23T02:05:25.6357063Z OK 2022-11-23T02:05:25.6357069Z 2022-11-23T02:05:25.6357180Z Generating XML reports... 2022-11-23T02:05:25.6357672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014418.xml 2022-11-23T02:05:25.6358008Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6358403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6358571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6358983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6359165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6359173Z 2022-11-23T02:05:25.6359324Z Running tests... 2022-11-23T02:05:25.6359603Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6359940Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11486 2022-11-23T02:05:25.6360156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11487 2022-11-23T02:05:25.6360427Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6360822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6360989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6361390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6361569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6361813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6362198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6362413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6362819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6362999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6363241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6363676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6364095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6364325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6364551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6364799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6365038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6365463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6365884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6366244Z STAGE:2022-11-23 01:44:28 11486:11486 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6366595Z STAGE:2022-11-23 01:44:28 11487:11487 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6366962Z STAGE:2022-11-23 01:44:28 11487:11487 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6367591Z STAGE:2022-11-23 01:44:28 11486:11486 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:44:28 11487:11487 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6367599Z 2022-11-23T02:05:25.6367972Z STAGE:2022-11-23 01:44:28 11486:11486 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6368320Z STAGE:2022-11-23 01:44:28 11487:11487 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6368673Z STAGE:2022-11-23 01:44:28 11486:11486 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6369028Z STAGE:2022-11-23 01:44:28 11487:11487 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6369448Z STAGE:2022-11-23 01:44:28 11487:11487 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6369804Z STAGE:2022-11-23 01:44:28 11486:11486 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6370176Z STAGE:2022-11-23 01:44:28 11486:11486 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6370262Z ok (4.050s) 2022-11-23T02:05:25.6370270Z 2022-11-23T02:05:25.6370551Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6370641Z Ran 1 test in 4.051s 2022-11-23T02:05:25.6370655Z 2022-11-23T02:05:25.6370723Z OK 2022-11-23T02:05:25.6370730Z 2022-11-23T02:05:25.6370841Z Generating XML reports... 2022-11-23T02:05:25.6371330Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014425.xml 2022-11-23T02:05:25.6371666Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6372069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6372239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6372691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6372879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6372886Z 2022-11-23T02:05:25.6372979Z Running tests... 2022-11-23T02:05:25.6373262Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6373604Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11681 2022-11-23T02:05:25.6373821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11682 2022-11-23T02:05:25.6374094Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6374492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6374659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6375073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6375256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6375498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6375931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6376320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6376492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6376897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6377071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6377311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6377733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6377962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6378184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6378422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6378661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6379137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6379564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6379927Z STAGE:2022-11-23 01:44:35 11681:11681 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6380277Z STAGE:2022-11-23 01:44:35 11682:11682 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6380640Z STAGE:2022-11-23 01:44:35 11681:11681 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6380989Z STAGE:2022-11-23 01:44:35 11682:11682 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6381362Z STAGE:2022-11-23 01:44:35 11681:11681 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6381738Z STAGE:2022-11-23 01:44:35 11682:11682 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6382088Z STAGE:2022-11-23 01:44:35 11681:11681 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6382476Z STAGE:2022-11-23 01:44:35 11682:11682 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6382839Z STAGE:2022-11-23 01:44:35 11682:11682 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6383186Z STAGE:2022-11-23 01:44:35 11681:11681 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6383555Z STAGE:2022-11-23 01:44:35 11682:11682 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6383923Z STAGE:2022-11-23 01:44:35 11681:11681 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6384009Z ok (3.876s) 2022-11-23T02:05:25.6384017Z 2022-11-23T02:05:25.6384301Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6384393Z Ran 1 test in 3.877s 2022-11-23T02:05:25.6384407Z 2022-11-23T02:05:25.6384476Z OK 2022-11-23T02:05:25.6384482Z 2022-11-23T02:05:25.6384593Z Generating XML reports... 2022-11-23T02:05:25.6385085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014433.xml 2022-11-23T02:05:25.6385417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6385817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6385987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6386394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6386581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6386591Z 2022-11-23T02:05:25.6386685Z Running tests... 2022-11-23T02:05:25.6386965Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6387300Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11876 2022-11-23T02:05:25.6387515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11877 2022-11-23T02:05:25.6387784Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6388176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6388344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6388748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6388980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6389225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6389615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6389780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6390182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6390366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6390605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6391034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6391469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6391691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6391957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6392197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6392432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6392860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6393284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6393646Z STAGE:2022-11-23 01:44:42 11876:11876 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6394008Z STAGE:2022-11-23 01:44:42 11877:11877 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6394368Z STAGE:2022-11-23 01:44:42 11877:11877 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6394988Z STAGE:2022-11-23 01:44:42 11876:11876 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:44:42 11877:11877 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6394996Z 2022-11-23T02:05:25.6395372Z STAGE:2022-11-23 01:44:42 11876:11876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6395724Z STAGE:2022-11-23 01:44:42 11877:11877 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6396124Z STAGE:2022-11-23 01:44:42 11876:11876 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6396477Z STAGE:2022-11-23 01:44:42 11877:11877 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6396852Z STAGE:2022-11-23 01:44:42 11877:11877 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6397210Z STAGE:2022-11-23 01:44:42 11876:11876 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6397578Z STAGE:2022-11-23 01:44:42 11876:11876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6397663Z ok (3.951s) 2022-11-23T02:05:25.6397670Z 2022-11-23T02:05:25.6397954Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6398050Z Ran 1 test in 3.951s 2022-11-23T02:05:25.6398057Z 2022-11-23T02:05:25.6398134Z OK 2022-11-23T02:05:25.6398140Z 2022-11-23T02:05:25.6398248Z Generating XML reports... 2022-11-23T02:05:25.6398736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014440.xml 2022-11-23T02:05:25.6399118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6399519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6399686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6400094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6400278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6400285Z 2022-11-23T02:05:25.6400379Z Running tests... 2022-11-23T02:05:25.6400659Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6400987Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12071 2022-11-23T02:05:25.6401203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12072 2022-11-23T02:05:25.6401476Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6401867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6402083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6402492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6402677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6402921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6403313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6403479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6403888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6404072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6404318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6404746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6405177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6405404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6405621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6405772Z skip: Skipped due to small world size. (3.738s) 2022-11-23T02:05:25.6405783Z 2022-11-23T02:05:25.6406062Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6406158Z Ran 1 test in 3.738s 2022-11-23T02:05:25.6406166Z 2022-11-23T02:05:25.6406255Z OK (skipped=1) 2022-11-23T02:05:25.6406261Z 2022-11-23T02:05:25.6406368Z Generating XML reports... 2022-11-23T02:05:25.6406856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014447.xml 2022-11-23T02:05:25.6407186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6407579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6407744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6408149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6408335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6408392Z 2022-11-23T02:05:25.6408484Z Running tests... 2022-11-23T02:05:25.6408767Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6409101Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12254 2022-11-23T02:05:25.6409316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12255 2022-11-23T02:05:25.6409588Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6409980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6410146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6410546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6410731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6410972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6411440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6411832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6411998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6412400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6412582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6412824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6413254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6413481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6413707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6413857Z skip: Skipped due to small world size. (3.877s) 2022-11-23T02:05:25.6413865Z 2022-11-23T02:05:25.6414140Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6414237Z Ran 1 test in 3.877s 2022-11-23T02:05:25.6414244Z 2022-11-23T02:05:25.6414334Z OK (skipped=1) 2022-11-23T02:05:25.6414341Z 2022-11-23T02:05:25.6414448Z Generating XML reports... 2022-11-23T02:05:25.6414934Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014453.xml 2022-11-23T02:05:25.6415264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6415662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6415829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6416235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6416418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6416425Z 2022-11-23T02:05:25.6416518Z Running tests... 2022-11-23T02:05:25.6416793Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6417130Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12437 2022-11-23T02:05:25.6417339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12438 2022-11-23T02:05:25.6417656Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6418052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6418221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6418618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6418797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6419038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6419462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6419851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6420019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6420421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6420666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6420910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6421337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6421560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6421786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6421935Z skip: Skipped due to small world size. (3.940s) 2022-11-23T02:05:25.6421943Z 2022-11-23T02:05:25.6422226Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6422322Z Ran 1 test in 3.940s 2022-11-23T02:05:25.6422330Z 2022-11-23T02:05:25.6422419Z OK (skipped=1) 2022-11-23T02:05:25.6422425Z 2022-11-23T02:05:25.6422534Z Generating XML reports... 2022-11-23T02:05:25.6423025Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014500.xml 2022-11-23T02:05:25.6423356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6423747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6423911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6424314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6424495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6424506Z 2022-11-23T02:05:25.6424598Z Running tests... 2022-11-23T02:05:25.6424877Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6425203Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12620 2022-11-23T02:05:25.6425419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12621 2022-11-23T02:05:25.6425686Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6426078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6426243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6426640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6426865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6427108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6427541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6427931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6428101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6428504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6428688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6428931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6429361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6429587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6429853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6430000Z skip: Skipped due to small world size. (3.738s) 2022-11-23T02:05:25.6430014Z 2022-11-23T02:05:25.6430290Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6430387Z Ran 1 test in 3.738s 2022-11-23T02:05:25.6430394Z 2022-11-23T02:05:25.6430484Z OK (skipped=1) 2022-11-23T02:05:25.6430491Z 2022-11-23T02:05:25.6430602Z Generating XML reports... 2022-11-23T02:05:25.6431091Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014507.xml 2022-11-23T02:05:25.6431423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6431821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6431986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6432393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6432576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6432583Z 2022-11-23T02:05:25.6432676Z Running tests... 2022-11-23T02:05:25.6432955Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6433278Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12803 2022-11-23T02:05:25.6433494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12804 2022-11-23T02:05:25.6433768Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6434158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6434326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6434733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6434914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6435157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6435588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6435974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6436238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6436649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6436833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6437078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6437506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6437738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6438099Z STAGE:2022-11-23 01:45:17 12804:12804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6438323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6438677Z STAGE:2022-11-23 01:45:17 12803:12803 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6439033Z STAGE:2022-11-23 01:45:17 12804:12804 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6439384Z STAGE:2022-11-23 01:45:17 12803:12803 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6439809Z STAGE:2022-11-23 01:45:17 12804:12804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6440185Z STAGE:2022-11-23 01:45:17 12803:12803 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6440538Z STAGE:2022-11-23 01:45:17 12804:12804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6440880Z STAGE:2022-11-23 01:45:17 12803:12803 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6441485Z STAGE:2022-11-23 01:45:17 12803:12803 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:17 12804:12804 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6441496Z 2022-11-23T02:05:25.6442128Z STAGE:2022-11-23 01:45:17 12803:12803 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:45:17 12804:12804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6442136Z 2022-11-23T02:05:25.6442222Z ok (3.848s) 2022-11-23T02:05:25.6442229Z 2022-11-23T02:05:25.6442510Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6442609Z Ran 1 test in 3.848s 2022-11-23T02:05:25.6442616Z 2022-11-23T02:05:25.6442689Z OK 2022-11-23T02:05:25.6442696Z 2022-11-23T02:05:25.6442806Z Generating XML reports... 2022-11-23T02:05:25.6443293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014514.xml 2022-11-23T02:05:25.6443625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6444024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6444194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6444595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6444782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6444789Z 2022-11-23T02:05:25.6444881Z Running tests... 2022-11-23T02:05:25.6445159Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6445475Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12992 2022-11-23T02:05:25.6445692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12993 2022-11-23T02:05:25.6445960Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6446404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6446570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6446979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6447161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6447405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6447797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6447966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6448367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6448553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6448799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6449274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6449708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6449935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6450300Z STAGE:2022-11-23 01:45:24 12993:12993 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6450531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6450892Z STAGE:2022-11-23 01:45:24 12992:12992 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6451249Z STAGE:2022-11-23 01:45:24 12992:12992 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6451625Z STAGE:2022-11-23 01:45:24 12992:12992 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6451988Z STAGE:2022-11-23 01:45:24 12993:12993 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6452381Z STAGE:2022-11-23 01:45:24 12993:12993 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6452628Z STAGE:2022-11-23 01:45:24 12992:12992 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6452872Z STAGE:2022-11-23 01:45:24 12993:12993 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6453293Z STAGE:2022-11-23 01:45:24 12992:12992 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:24 12993:12993 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6453301Z 2022-11-23T02:05:25.6453744Z STAGE:2022-11-23 01:45:24 12992:12992 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:45:24 12993:12993 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6453752Z 2022-11-23T02:05:25.6453813Z ok (3.934s) 2022-11-23T02:05:25.6453818Z 2022-11-23T02:05:25.6454015Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6454083Z Ran 1 test in 3.934s 2022-11-23T02:05:25.6454088Z 2022-11-23T02:05:25.6454139Z OK 2022-11-23T02:05:25.6454143Z 2022-11-23T02:05:25.6454218Z Generating XML reports... 2022-11-23T02:05:25.6454562Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014521.xml 2022-11-23T02:05:25.6454799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6455116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6455234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6455523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6455654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6455659Z 2022-11-23T02:05:25.6455724Z Running tests... 2022-11-23T02:05:25.6455922Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6456153Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13181 2022-11-23T02:05:25.6456308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13182 2022-11-23T02:05:25.6456499Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6456785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6456902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6457217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6457344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6457517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6457792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6457909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6458187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6458318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6458485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6458787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6459084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6459243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6459399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6459656Z STAGE:2022-11-23 01:45:31 13182:13182 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6459905Z STAGE:2022-11-23 01:45:31 13181:13181 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6460163Z STAGE:2022-11-23 01:45:31 13181:13181 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6460598Z STAGE:2022-11-23 01:45:31 13182:13182 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:31 13181:13181 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6460604Z 2022-11-23T02:05:25.6460771Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6461038Z STAGE:2022-11-23 01:45:31 13182:13182 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6461206Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6461456Z STAGE:2022-11-23 01:45:31 13181:13181 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6461700Z STAGE:2022-11-23 01:45:31 13182:13182 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6461989Z STAGE:2022-11-23 01:45:31 13182:13182 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6462248Z STAGE:2022-11-23 01:45:31 13182:13182 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6462502Z STAGE:2022-11-23 01:45:31 13181:13181 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6462761Z STAGE:2022-11-23 01:45:31 13181:13181 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6463361Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:05:25.6463431Z warnings.warn( 2022-11-23T02:05:25.6464027Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:05:25.6464095Z warnings.warn( 2022-11-23T02:05:25.6464182Z ok (4.242s) 2022-11-23T02:05:25.6464187Z 2022-11-23T02:05:25.6464389Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6464458Z Ran 1 test in 4.242s 2022-11-23T02:05:25.6464463Z 2022-11-23T02:05:25.6464516Z OK 2022-11-23T02:05:25.6464520Z 2022-11-23T02:05:25.6464599Z Generating XML reports... 2022-11-23T02:05:25.6464941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014528.xml 2022-11-23T02:05:25.6465179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6465458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6465578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6465861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6465994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6465999Z 2022-11-23T02:05:25.6466065Z Running tests... 2022-11-23T02:05:25.6466261Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6466502Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13370 2022-11-23T02:05:25.6466654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13371 2022-11-23T02:05:25.6466844Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6467123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6467240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6467527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6467657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6467830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6468105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6468218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6468497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6468661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6468832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6469139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6469439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6469602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6469761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6470014Z STAGE:2022-11-23 01:45:38 13371:13371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6470260Z STAGE:2022-11-23 01:45:38 13370:13370 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6470515Z STAGE:2022-11-23 01:45:38 13370:13370 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6470776Z STAGE:2022-11-23 01:45:38 13370:13370 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6470981Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6471233Z STAGE:2022-11-23 01:45:38 13371:13371 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6471497Z STAGE:2022-11-23 01:45:38 13371:13371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6471663Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6471907Z STAGE:2022-11-23 01:45:38 13370:13370 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6472148Z STAGE:2022-11-23 01:45:38 13371:13371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6472399Z STAGE:2022-11-23 01:45:38 13370:13370 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6472658Z STAGE:2022-11-23 01:45:38 13370:13370 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6472913Z STAGE:2022-11-23 01:45:38 13371:13371 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6473174Z STAGE:2022-11-23 01:45:38 13371:13371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6473766Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:05:25.6473834Z warnings.warn( 2022-11-23T02:05:25.6474412Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:05:25.6474484Z warnings.warn( 2022-11-23T02:05:25.6474538Z ok (4.138s) 2022-11-23T02:05:25.6474548Z 2022-11-23T02:05:25.6474745Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6474814Z Ran 1 test in 4.139s 2022-11-23T02:05:25.6474820Z 2022-11-23T02:05:25.6474872Z OK 2022-11-23T02:05:25.6474878Z 2022-11-23T02:05:25.6474955Z Generating XML reports... 2022-11-23T02:05:25.6475298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014535.xml 2022-11-23T02:05:25.6475532Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6475810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6475961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6476302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6476438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6476444Z 2022-11-23T02:05:25.6476509Z Running tests... 2022-11-23T02:05:25.6476707Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6476940Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13559 2022-11-23T02:05:25.6477091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13560 2022-11-23T02:05:25.6477280Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6477558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6477676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6478001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6478132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6478307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6478585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6478697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6478984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6479117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6479290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6479593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6479894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6480056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6480311Z STAGE:2022-11-23 01:45:45 13560:13560 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6480468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6480719Z STAGE:2022-11-23 01:45:45 13559:13559 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6480974Z STAGE:2022-11-23 01:45:45 13559:13559 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6481406Z STAGE:2022-11-23 01:45:45 13560:13560 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:45 13559:13559 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6481413Z 2022-11-23T02:05:25.6481678Z STAGE:2022-11-23 01:45:45 13560:13560 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6481923Z STAGE:2022-11-23 01:45:45 13560:13560 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6482163Z STAGE:2022-11-23 01:45:45 13559:13559 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6482583Z STAGE:2022-11-23 01:45:45 13559:13559 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:45 13560:13560 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6482658Z 2022-11-23T02:05:25.6483100Z STAGE:2022-11-23 01:45:45 13559:13559 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:45:45 13560:13560 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6483106Z 2022-11-23T02:05:25.6483165Z ok (3.737s) 2022-11-23T02:05:25.6483170Z 2022-11-23T02:05:25.6483368Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6483438Z Ran 1 test in 3.738s 2022-11-23T02:05:25.6483443Z 2022-11-23T02:05:25.6483494Z OK 2022-11-23T02:05:25.6483499Z 2022-11-23T02:05:25.6483575Z Generating XML reports... 2022-11-23T02:05:25.6483917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014543.xml 2022-11-23T02:05:25.6484153Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6484433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6484553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6484841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6485004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6485009Z 2022-11-23T02:05:25.6485076Z Running tests... 2022-11-23T02:05:25.6485268Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6485500Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13748 2022-11-23T02:05:25.6485654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13749 2022-11-23T02:05:25.6485845Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6486123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6486240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6486526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6486658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6486829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6487101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6487218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6487503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6487631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6487805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6488106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6488409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6488568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6488722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6488783Z ok (4.451s) 2022-11-23T02:05:25.6488788Z 2022-11-23T02:05:25.6488986Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6489056Z Ran 1 test in 4.451s 2022-11-23T02:05:25.6489061Z 2022-11-23T02:05:25.6489115Z OK 2022-11-23T02:05:25.6489151Z 2022-11-23T02:05:25.6489223Z Generating XML reports... 2022-11-23T02:05:25.6489567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014550.xml 2022-11-23T02:05:25.6489802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6490084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6490203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6490487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6490618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6490623Z 2022-11-23T02:05:25.6490689Z Running tests... 2022-11-23T02:05:25.6490886Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6491112Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13931 2022-11-23T02:05:25.6491265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13932 2022-11-23T02:05:25.6491486Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6491766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6491886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6492168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6492297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6492471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6492745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6492863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6493149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6493281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6493452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6493753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6494049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6494209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6494370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6494628Z STAGE:2022-11-23 01:45:59 13931:13931 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6494876Z STAGE:2022-11-23 01:45:59 13932:13932 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6495300Z STAGE:2022-11-23 01:45:59 13932:13932 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:59 13931:13931 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6495306Z 2022-11-23T02:05:25.6495751Z STAGE:2022-11-23 01:45:59 13932:13932 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:45:59 13931:13931 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6495756Z 2022-11-23T02:05:25.6496005Z STAGE:2022-11-23 01:45:59 13931:13931 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6496287Z STAGE:2022-11-23 01:45:59 13932:13932 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6496539Z STAGE:2022-11-23 01:45:59 13932:13932 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6496802Z STAGE:2022-11-23 01:45:59 13932:13932 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6497053Z STAGE:2022-11-23 01:45:59 13931:13931 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6497312Z STAGE:2022-11-23 01:45:59 13931:13931 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6497374Z ok (4.038s) 2022-11-23T02:05:25.6497379Z 2022-11-23T02:05:25.6497580Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6497648Z Ran 1 test in 4.039s 2022-11-23T02:05:25.6497653Z 2022-11-23T02:05:25.6497706Z OK 2022-11-23T02:05:25.6497713Z 2022-11-23T02:05:25.6497789Z Generating XML reports... 2022-11-23T02:05:25.6498130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014557.xml 2022-11-23T02:05:25.6498366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6498677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6498800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6499088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6499219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6499224Z 2022-11-23T02:05:25.6499291Z Running tests... 2022-11-23T02:05:25.6499490Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6499720Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14120 2022-11-23T02:05:25.6499873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14121 2022-11-23T02:05:25.6500066Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6500343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6500461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6500747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6500876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6501049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6501354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6501630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6501749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6502031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6502157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6502327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6502627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6502785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6503078Z STAGE:2022-11-23 01:46:06 14121:14121 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6503237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6503490Z STAGE:2022-11-23 01:46:07 14120:14120 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6503744Z STAGE:2022-11-23 01:46:07 14120:14120 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6503990Z STAGE:2022-11-23 01:46:07 14121:14121 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6504255Z STAGE:2022-11-23 01:46:07 14120:14120 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6504517Z STAGE:2022-11-23 01:46:07 14121:14121 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6504932Z STAGE:2022-11-23 01:46:07 14121:14121 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:46:07 14120:14120 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6504940Z 2022-11-23T02:05:25.6505193Z STAGE:2022-11-23 01:46:07 14121:14121 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6505472Z STAGE:2022-11-23 01:46:07 14120:14120 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6505731Z STAGE:2022-11-23 01:46:07 14121:14121 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6505991Z STAGE:2022-11-23 01:46:07 14120:14120 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6506052Z ok (3.947s) 2022-11-23T02:05:25.6506057Z 2022-11-23T02:05:25.6506253Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6506324Z Ran 1 test in 3.947s 2022-11-23T02:05:25.6506329Z 2022-11-23T02:05:25.6506383Z OK 2022-11-23T02:05:25.6506388Z 2022-11-23T02:05:25.6506467Z Generating XML reports... 2022-11-23T02:05:25.6506811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014604.xml 2022-11-23T02:05:25.6507042Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6507322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6507440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6507723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6507852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6507858Z 2022-11-23T02:05:25.6507923Z Running tests... 2022-11-23T02:05:25.6508124Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6508362Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14309 2022-11-23T02:05:25.6508517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14310 2022-11-23T02:05:25.6508712Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6508989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6509107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6509393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6509522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6509688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6510030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6510306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6510428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6510711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6510838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6511006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6511304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6511463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6511719Z STAGE:2022-11-23 01:46:13 14310:14310 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6511877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6512158Z STAGE:2022-11-23 01:46:13 14309:14309 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6512415Z STAGE:2022-11-23 01:46:13 14309:14309 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6512663Z STAGE:2022-11-23 01:46:13 14310:14310 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6512925Z STAGE:2022-11-23 01:46:13 14309:14309 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6513189Z STAGE:2022-11-23 01:46:13 14310:14310 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6513434Z STAGE:2022-11-23 01:46:13 14309:14309 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6513685Z STAGE:2022-11-23 01:46:13 14310:14310 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6514110Z STAGE:2022-11-23 01:46:13 14310:14310 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:46:13 14309:14309 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6514116Z 2022-11-23T02:05:25.6514555Z STAGE:2022-11-23 01:46:13 14310:14310 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:46:13 14309:14309 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6514561Z 2022-11-23T02:05:25.6514624Z ok (3.838s) 2022-11-23T02:05:25.6514629Z 2022-11-23T02:05:25.6514829Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6514899Z Ran 1 test in 3.839s 2022-11-23T02:05:25.6514904Z 2022-11-23T02:05:25.6514955Z OK 2022-11-23T02:05:25.6514960Z 2022-11-23T02:05:25.6515039Z Generating XML reports... 2022-11-23T02:05:25.6515379Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014611.xml 2022-11-23T02:05:25.6515614Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6515889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6516009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6516338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6516468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6516473Z 2022-11-23T02:05:25.6516539Z Running tests... 2022-11-23T02:05:25.6516900Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6517144Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14498 2022-11-23T02:05:25.6517442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14499 2022-11-23T02:05:25.6517646Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6517943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6518070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6518374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6518512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6518692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6518989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6519114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6519418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6519591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6519776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6520095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6520413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6520580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6520749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6521014Z STAGE:2022-11-23 01:46:21 14499:14499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6521277Z STAGE:2022-11-23 01:46:21 14498:14498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6521546Z STAGE:2022-11-23 01:46:21 14498:14498 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6521825Z STAGE:2022-11-23 01:46:21 14498:14498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6522258Z STAGE:2022-11-23 01:46:21 14499:14499 ActivityProfilerController.cpp:306] Completed Stage: Collection[W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6522264Z 2022-11-23T02:05:25.6522543Z STAGE:2022-11-23 01:46:21 14499:14499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6522719Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6522982Z STAGE:2022-11-23 01:46:21 14498:14498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6523239Z STAGE:2022-11-23 01:46:21 14499:14499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6523504Z STAGE:2022-11-23 01:46:21 14499:14499 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6523776Z STAGE:2022-11-23 01:46:21 14499:14499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6524043Z STAGE:2022-11-23 01:46:21 14498:14498 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6524319Z STAGE:2022-11-23 01:46:21 14498:14498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6524582Z STAGE:2022-11-23 01:46:21 14499:14499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6524873Z STAGE:2022-11-23 01:46:21 14498:14498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6525137Z STAGE:2022-11-23 01:46:21 14499:14499 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6525413Z STAGE:2022-11-23 01:46:21 14499:14499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6525678Z STAGE:2022-11-23 01:46:21 14498:14498 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6525951Z STAGE:2022-11-23 01:46:21 14498:14498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6526016Z ok (4.451s) 2022-11-23T02:05:25.6526022Z 2022-11-23T02:05:25.6526231Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6526304Z Ran 1 test in 4.452s 2022-11-23T02:05:25.6526310Z 2022-11-23T02:05:25.6526365Z OK 2022-11-23T02:05:25.6526370Z 2022-11-23T02:05:25.6526453Z Generating XML reports... 2022-11-23T02:05:25.6526814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014618.xml 2022-11-23T02:05:25.6527056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6527384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6527510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6527917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6528049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6528053Z 2022-11-23T02:05:25.6528119Z Running tests... 2022-11-23T02:05:25.6528317Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6528550Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14687 2022-11-23T02:05:25.6528705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14688 2022-11-23T02:05:25.6528898Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6529174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6529290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6529576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6529703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6529877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6530157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6530274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6530559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6530691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6530863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6531162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6531463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6531620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6531800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6532056Z STAGE:2022-11-23 01:46:28 14688:14688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6532304Z STAGE:2022-11-23 01:46:28 14687:14687 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6532562Z STAGE:2022-11-23 01:46:28 14687:14687 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6532825Z STAGE:2022-11-23 01:46:28 14687:14687 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6532990Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6533241Z STAGE:2022-11-23 01:46:28 14688:14688 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6533503Z STAGE:2022-11-23 01:46:28 14688:14688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6533669Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6533916Z STAGE:2022-11-23 01:46:28 14687:14687 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6534209Z STAGE:2022-11-23 01:46:28 14688:14688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6534633Z STAGE:2022-11-23 01:46:29 14687:14687 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:46:29 14688:14688 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6534639Z 2022-11-23T02:05:25.6534901Z STAGE:2022-11-23 01:46:29 14687:14687 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6535160Z STAGE:2022-11-23 01:46:29 14688:14688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6535407Z STAGE:2022-11-23 01:46:29 14687:14687 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6535650Z STAGE:2022-11-23 01:46:29 14688:14688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6535901Z STAGE:2022-11-23 01:46:29 14688:14688 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6536148Z STAGE:2022-11-23 01:46:29 14687:14687 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6536409Z STAGE:2022-11-23 01:46:29 14688:14688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6536668Z STAGE:2022-11-23 01:46:29 14687:14687 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6536729Z ok (4.447s) 2022-11-23T02:05:25.6536734Z 2022-11-23T02:05:25.6536933Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6537002Z Ran 1 test in 4.448s 2022-11-23T02:05:25.6537006Z 2022-11-23T02:05:25.6537060Z OK 2022-11-23T02:05:25.6537064Z 2022-11-23T02:05:25.6537144Z Generating XML reports... 2022-11-23T02:05:25.6537483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014626.xml 2022-11-23T02:05:25.6537716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6537993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6538113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6538396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6538525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6538530Z 2022-11-23T02:05:25.6538594Z Running tests... 2022-11-23T02:05:25.6538792Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6539032Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14876 2022-11-23T02:05:25.6539223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14877 2022-11-23T02:05:25.6539416Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6539697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6539817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6540103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6540233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6540406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6540686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6540803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6541119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6541248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6541418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6541723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6542026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6542187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6542343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6542596Z STAGE:2022-11-23 01:46:36 14877:14877 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6542847Z STAGE:2022-11-23 01:46:36 14876:14876 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6543099Z STAGE:2022-11-23 01:46:36 14877:14877 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6543346Z STAGE:2022-11-23 01:46:36 14876:14876 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6543606Z STAGE:2022-11-23 01:46:36 14876:14876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6543867Z STAGE:2022-11-23 01:46:36 14877:14877 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6544033Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6544200Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:05:25.6544449Z STAGE:2022-11-23 01:46:36 14876:14876 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6544695Z STAGE:2022-11-23 01:46:36 14877:14877 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6544946Z STAGE:2022-11-23 01:46:36 14877:14877 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6545207Z STAGE:2022-11-23 01:46:36 14877:14877 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6545455Z STAGE:2022-11-23 01:46:36 14876:14876 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6545713Z STAGE:2022-11-23 01:46:36 14876:14876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6545959Z STAGE:2022-11-23 01:46:36 14877:14877 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6546235Z STAGE:2022-11-23 01:46:36 14876:14876 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6546485Z STAGE:2022-11-23 01:46:36 14877:14877 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6546731Z STAGE:2022-11-23 01:46:36 14876:14876 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6546995Z STAGE:2022-11-23 01:46:36 14877:14877 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6547257Z STAGE:2022-11-23 01:46:36 14876:14876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6547318Z ok (4.346s) 2022-11-23T02:05:25.6547323Z 2022-11-23T02:05:25.6547519Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6547582Z Ran 1 test in 4.347s 2022-11-23T02:05:25.6547592Z 2022-11-23T02:05:25.6547640Z OK 2022-11-23T02:05:25.6547644Z 2022-11-23T02:05:25.6547726Z Generating XML reports... 2022-11-23T02:05:25.6548072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014633.xml 2022-11-23T02:05:25.6548305Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6548618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6548737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6549023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6549154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6549159Z 2022-11-23T02:05:25.6549226Z Running tests... 2022-11-23T02:05:25.6549420Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6549592Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:05:25.6549600Z 2022-11-23T02:05:25.6549789Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6549858Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6549863Z 2022-11-23T02:05:25.6549927Z OK (skipped=1) 2022-11-23T02:05:25.6549932Z 2022-11-23T02:05:25.6550014Z Generating XML reports... 2022-11-23T02:05:25.6550353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014641.xml 2022-11-23T02:05:25.6550588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6550869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6550988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6551271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6551401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6551406Z 2022-11-23T02:05:25.6551466Z Running tests... 2022-11-23T02:05:25.6551664Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6551848Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:05:25.6551853Z 2022-11-23T02:05:25.6552047Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6552115Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6552119Z 2022-11-23T02:05:25.6552183Z OK (skipped=1) 2022-11-23T02:05:25.6552187Z 2022-11-23T02:05:25.6552265Z Generating XML reports... 2022-11-23T02:05:25.6552604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014644.xml 2022-11-23T02:05:25.6552836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6553147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6553263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6553550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6553678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6553684Z 2022-11-23T02:05:25.6553748Z Running tests... 2022-11-23T02:05:25.6553945Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6554131Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s) 2022-11-23T02:05:25.6554136Z 2022-11-23T02:05:25.6554331Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6554402Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6554407Z 2022-11-23T02:05:25.6554471Z OK (skipped=1) 2022-11-23T02:05:25.6554475Z 2022-11-23T02:05:25.6554555Z Generating XML reports... 2022-11-23T02:05:25.6554895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014646.xml 2022-11-23T02:05:25.6555160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6555440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6555553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6555838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6555969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6555974Z 2022-11-23T02:05:25.6556039Z Running tests... 2022-11-23T02:05:25.6556289Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6556481Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s) 2022-11-23T02:05:25.6556486Z 2022-11-23T02:05:25.6556848Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6556922Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6556927Z 2022-11-23T02:05:25.6556993Z OK (skipped=1) 2022-11-23T02:05:25.6556998Z 2022-11-23T02:05:25.6557077Z Generating XML reports... 2022-11-23T02:05:25.6557439Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014649.xml 2022-11-23T02:05:25.6557684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6557977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6558106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6558411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6558548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6558553Z 2022-11-23T02:05:25.6558624Z Running tests... 2022-11-23T02:05:25.6558833Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6559028Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:05:25.6559033Z 2022-11-23T02:05:25.6559240Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6559317Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6559322Z 2022-11-23T02:05:25.6559392Z OK (skipped=1) 2022-11-23T02:05:25.6559397Z 2022-11-23T02:05:25.6559471Z Generating XML reports... 2022-11-23T02:05:25.6559877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014652.xml 2022-11-23T02:05:25.6560124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6560419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6560543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6560844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6560981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6560987Z 2022-11-23T02:05:25.6561057Z Running tests... 2022-11-23T02:05:25.6561266Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6561475Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s) 2022-11-23T02:05:25.6561482Z 2022-11-23T02:05:25.6561685Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6561756Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6561761Z 2022-11-23T02:05:25.6561829Z OK (skipped=1) 2022-11-23T02:05:25.6561834Z 2022-11-23T02:05:25.6561950Z Generating XML reports... 2022-11-23T02:05:25.6562313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014655.xml 2022-11-23T02:05:25.6562561Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6562853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6562978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6563278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6563418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6563423Z 2022-11-23T02:05:25.6563495Z Running tests... 2022-11-23T02:05:25.6563697Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6563889Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:05:25.6563894Z 2022-11-23T02:05:25.6564091Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6564164Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6564169Z 2022-11-23T02:05:25.6564236Z OK (skipped=1) 2022-11-23T02:05:25.6564240Z 2022-11-23T02:05:25.6564321Z Generating XML reports... 2022-11-23T02:05:25.6564682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014658.xml 2022-11-23T02:05:25.6564932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6565231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6565356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6565659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6565796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6565801Z 2022-11-23T02:05:25.6565869Z Running tests... 2022-11-23T02:05:25.6566078Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6566288Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s) 2022-11-23T02:05:25.6566293Z 2022-11-23T02:05:25.6566498Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6566571Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6566610Z 2022-11-23T02:05:25.6566681Z OK (skipped=1) 2022-11-23T02:05:25.6566685Z 2022-11-23T02:05:25.6566769Z Generating XML reports... 2022-11-23T02:05:25.6567230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014701.xml 2022-11-23T02:05:25.6567467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6567747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6567866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6568151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6568281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6568286Z 2022-11-23T02:05:25.6568347Z Running tests... 2022-11-23T02:05:25.6568546Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6568750Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:05:25.6568755Z 2022-11-23T02:05:25.6568949Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6569052Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6569057Z 2022-11-23T02:05:25.6569125Z OK (skipped=1) 2022-11-23T02:05:25.6569129Z 2022-11-23T02:05:25.6569206Z Generating XML reports... 2022-11-23T02:05:25.6569551Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014704.xml 2022-11-23T02:05:25.6569785Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6570061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6570331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6570617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6570746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6570751Z 2022-11-23T02:05:25.6570821Z Running tests... 2022-11-23T02:05:25.6571019Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6571233Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:05:25.6571238Z 2022-11-23T02:05:25.6571433Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6571503Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6571507Z 2022-11-23T02:05:25.6571571Z OK (skipped=1) 2022-11-23T02:05:25.6571576Z 2022-11-23T02:05:25.6571653Z Generating XML reports... 2022-11-23T02:05:25.6571993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014707.xml 2022-11-23T02:05:25.6572229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6572511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6572625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6572911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6573042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6573047Z 2022-11-23T02:05:25.6573114Z Running tests... 2022-11-23T02:05:25.6573311Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6573521Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:05:25.6573563Z 2022-11-23T02:05:25.6573758Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6573829Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6573834Z 2022-11-23T02:05:25.6573897Z OK (skipped=1) 2022-11-23T02:05:25.6573902Z 2022-11-23T02:05:25.6573980Z Generating XML reports... 2022-11-23T02:05:25.6574318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014710.xml 2022-11-23T02:05:25.6574550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6574826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6574948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6575232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6575364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6575369Z 2022-11-23T02:05:25.6575434Z Running tests... 2022-11-23T02:05:25.6575631Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6575884Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:05:25.6575890Z 2022-11-23T02:05:25.6576087Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6576157Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6576162Z 2022-11-23T02:05:25.6576227Z OK (skipped=1) 2022-11-23T02:05:25.6576232Z 2022-11-23T02:05:25.6576309Z Generating XML reports... 2022-11-23T02:05:25.6576639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014713.xml 2022-11-23T02:05:25.6576871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6577151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6577268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6577554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6577686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6577690Z 2022-11-23T02:05:25.6577759Z Running tests... 2022-11-23T02:05:25.6577952Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6578164Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:05:25.6578169Z 2022-11-23T02:05:25.6578358Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6578428Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6578434Z 2022-11-23T02:05:25.6578498Z OK (skipped=1) 2022-11-23T02:05:25.6578503Z 2022-11-23T02:05:25.6578579Z Generating XML reports... 2022-11-23T02:05:25.6578917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014716.xml 2022-11-23T02:05:25.6579150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6579430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6579548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6579834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6579962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6579967Z 2022-11-23T02:05:25.6580066Z Running tests... 2022-11-23T02:05:25.6580259Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6580480Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s) 2022-11-23T02:05:25.6580486Z 2022-11-23T02:05:25.6580681Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6580745Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6580754Z 2022-11-23T02:05:25.6580812Z OK (skipped=1) 2022-11-23T02:05:25.6580823Z 2022-11-23T02:05:25.6580895Z Generating XML reports... 2022-11-23T02:05:25.6581234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014718.xml 2022-11-23T02:05:25.6581467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6581744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6581865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6582149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6582310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6582315Z 2022-11-23T02:05:25.6582380Z Running tests... 2022-11-23T02:05:25.6582577Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6582786Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:05:25.6582791Z 2022-11-23T02:05:25.6582986Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6583055Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6583060Z 2022-11-23T02:05:25.6583124Z OK (skipped=1) 2022-11-23T02:05:25.6583131Z 2022-11-23T02:05:25.6583208Z Generating XML reports... 2022-11-23T02:05:25.6583547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014721.xml 2022-11-23T02:05:25.6583783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6584069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6584190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6584479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6584611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6584616Z 2022-11-23T02:05:25.6584685Z Running tests... 2022-11-23T02:05:25.6584874Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6585095Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s) 2022-11-23T02:05:25.6585109Z 2022-11-23T02:05:25.6585298Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6585370Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6585375Z 2022-11-23T02:05:25.6585443Z OK (skipped=1) 2022-11-23T02:05:25.6585448Z 2022-11-23T02:05:25.6585527Z Generating XML reports... 2022-11-23T02:05:25.6585869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014724.xml 2022-11-23T02:05:25.6586108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6586386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6586507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6586844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6586976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6586982Z 2022-11-23T02:05:25.6587048Z Running tests... 2022-11-23T02:05:25.6587247Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6587453Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:05:25.6587458Z 2022-11-23T02:05:25.6587652Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6587726Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6587730Z 2022-11-23T02:05:25.6587799Z OK (skipped=1) 2022-11-23T02:05:25.6587803Z 2022-11-23T02:05:25.6587886Z Generating XML reports... 2022-11-23T02:05:25.6588229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014727.xml 2022-11-23T02:05:25.6588466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6588744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6588899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6589188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6589311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6589324Z 2022-11-23T02:05:25.6589384Z Running tests... 2022-11-23T02:05:25.6589583Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6589799Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-11-23T02:05:25.6589803Z 2022-11-23T02:05:25.6590001Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6590072Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6590077Z 2022-11-23T02:05:25.6590142Z OK (skipped=1) 2022-11-23T02:05:25.6590147Z 2022-11-23T02:05:25.6590227Z Generating XML reports... 2022-11-23T02:05:25.6590568Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014730.xml 2022-11-23T02:05:25.6590804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6591082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6591200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6591486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6591615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6591622Z 2022-11-23T02:05:25.6591689Z Running tests... 2022-11-23T02:05:25.6591886Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6592101Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:05:25.6592106Z 2022-11-23T02:05:25.6592304Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6592374Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6592379Z 2022-11-23T02:05:25.6592447Z OK (skipped=1) 2022-11-23T02:05:25.6592452Z 2022-11-23T02:05:25.6592532Z Generating XML reports... 2022-11-23T02:05:25.6592878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014733.xml 2022-11-23T02:05:25.6593115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6593421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6593541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6593828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6593960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6593965Z 2022-11-23T02:05:25.6594030Z Running tests... 2022-11-23T02:05:25.6594231Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6594452Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:05:25.6594457Z 2022-11-23T02:05:25.6594652Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6594722Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6594729Z 2022-11-23T02:05:25.6594795Z OK (skipped=1) 2022-11-23T02:05:25.6594799Z 2022-11-23T02:05:25.6594877Z Generating XML reports... 2022-11-23T02:05:25.6595215Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014736.xml 2022-11-23T02:05:25.6595478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6595758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6595875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6596214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6596344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6596350Z 2022-11-23T02:05:25.6596415Z Running tests... 2022-11-23T02:05:25.6596615Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6596834Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:05:25.6596839Z 2022-11-23T02:05:25.6597030Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6597101Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6597106Z 2022-11-23T02:05:25.6597172Z OK (skipped=1) 2022-11-23T02:05:25.6597177Z 2022-11-23T02:05:25.6597248Z Generating XML reports... 2022-11-23T02:05:25.6597589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014739.xml 2022-11-23T02:05:25.6597823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6598101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6598219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6598554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6611685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6611700Z 2022-11-23T02:05:25.6611804Z Running tests... 2022-11-23T02:05:25.6612061Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6612294Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:05:25.6612300Z 2022-11-23T02:05:25.6612496Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6612566Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6612571Z 2022-11-23T02:05:25.6612634Z OK (skipped=1) 2022-11-23T02:05:25.6612638Z 2022-11-23T02:05:25.6612715Z Generating XML reports... 2022-11-23T02:05:25.6613181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014742.xml 2022-11-23T02:05:25.6613417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6613701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6613821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6614108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6614236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6614242Z 2022-11-23T02:05:25.6614306Z Running tests... 2022-11-23T02:05:25.6614502Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6614713Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:05:25.6614721Z 2022-11-23T02:05:25.6614914Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6614981Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6614986Z 2022-11-23T02:05:25.6615047Z OK (skipped=1) 2022-11-23T02:05:25.6615052Z 2022-11-23T02:05:25.6615170Z Generating XML reports... 2022-11-23T02:05:25.6615518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014745.xml 2022-11-23T02:05:25.6615749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6616024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6616140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6616423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6616553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6616558Z 2022-11-23T02:05:25.6616621Z Running tests... 2022-11-23T02:05:25.6616814Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6617032Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:05:25.6617038Z 2022-11-23T02:05:25.6617229Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6617292Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6617300Z 2022-11-23T02:05:25.6617358Z OK (skipped=1) 2022-11-23T02:05:25.6617362Z 2022-11-23T02:05:25.6617436Z Generating XML reports... 2022-11-23T02:05:25.6617779Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014748.xml 2022-11-23T02:05:25.6618010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6618290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6618404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6618689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6618816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6618822Z 2022-11-23T02:05:25.6618884Z Running tests... 2022-11-23T02:05:25.6619076Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6619308Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16457 2022-11-23T02:05:25.6619462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16458 2022-11-23T02:05:25.6619691Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6619970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6620089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6620373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6620503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6620673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6620975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6621249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6621370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6621656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6621813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6621989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6622293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6622452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6622614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6622783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6622956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6623255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6623553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6623613Z ok (4.641s) 2022-11-23T02:05:25.6623619Z 2022-11-23T02:05:25.6623817Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6623889Z Ran 1 test in 4.641s 2022-11-23T02:05:25.6623894Z 2022-11-23T02:05:25.6623947Z OK 2022-11-23T02:05:25.6623951Z 2022-11-23T02:05:25.6624029Z Generating XML reports... 2022-11-23T02:05:25.6624371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014751.xml 2022-11-23T02:05:25.6624607Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6624889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6625006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6625286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6625419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6625430Z 2022-11-23T02:05:25.6625491Z Running tests... 2022-11-23T02:05:25.6625688Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6626016Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s) 2022-11-23T02:05:25.6626021Z 2022-11-23T02:05:25.6626217Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6626322Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6626327Z 2022-11-23T02:05:25.6626391Z OK (skipped=1) 2022-11-23T02:05:25.6626395Z 2022-11-23T02:05:25.6626475Z Generating XML reports... 2022-11-23T02:05:25.6626815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014759.xml 2022-11-23T02:05:25.6627051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6627329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6627447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6627729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6627857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6627863Z 2022-11-23T02:05:25.6627929Z Running tests... 2022-11-23T02:05:25.6628123Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6628440Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:05:25.6628445Z 2022-11-23T02:05:25.6628698Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6628769Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6628773Z 2022-11-23T02:05:25.6628838Z OK (skipped=1) 2022-11-23T02:05:25.6628842Z 2022-11-23T02:05:25.6628920Z Generating XML reports... 2022-11-23T02:05:25.6629265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014802.xml 2022-11-23T02:05:25.6629502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6629777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6629898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6630181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6630314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6630319Z 2022-11-23T02:05:25.6630384Z Running tests... 2022-11-23T02:05:25.6630583Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6630803Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16769 2022-11-23T02:05:25.6630955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16770 2022-11-23T02:05:25.6631146Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6631421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6631542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6631829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6631959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6632132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6632438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6632715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6632832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6633117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6633280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6633453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6633756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6633917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6634076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6634135Z ok (4.659s) 2022-11-23T02:05:25.6634140Z 2022-11-23T02:05:25.6634336Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6634405Z Ran 1 test in 4.659s 2022-11-23T02:05:25.6634410Z 2022-11-23T02:05:25.6634462Z OK 2022-11-23T02:05:25.6634469Z 2022-11-23T02:05:25.6634545Z Generating XML reports... 2022-11-23T02:05:25.6634889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014805.xml 2022-11-23T02:05:25.6635122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6635430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6635550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6635834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6635962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6635967Z 2022-11-23T02:05:25.6636032Z Running tests... 2022-11-23T02:05:25.6636459Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6636697Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16952 2022-11-23T02:05:25.6636854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16953 2022-11-23T02:05:25.6637058Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6637349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6637473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6637773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6637908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6638088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6638377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6638501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6638804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6638940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6639120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6639438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6639754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6639922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6640131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6640197Z ok (5.091s) 2022-11-23T02:05:25.6640202Z 2022-11-23T02:05:25.6640412Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6640483Z Ran 1 test in 5.091s 2022-11-23T02:05:25.6640491Z 2022-11-23T02:05:25.6640549Z OK 2022-11-23T02:05:25.6640553Z 2022-11-23T02:05:25.6640634Z Generating XML reports... 2022-11-23T02:05:25.6640997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014813.xml 2022-11-23T02:05:25.6641243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6641541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6641664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6641970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6642107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6642112Z 2022-11-23T02:05:25.6642186Z Running tests... 2022-11-23T02:05:25.6642438Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6642675Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17135 2022-11-23T02:05:25.6642836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17136 2022-11-23T02:05:25.6643041Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6643333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6643462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6643760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6643894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6644076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6644399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6644693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6644816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6645113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6645248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6645432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6645750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6645920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6646088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6646265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6646444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6646761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6647163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6647257Z ok (4.657s) 2022-11-23T02:05:25.6647262Z 2022-11-23T02:05:25.6647463Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6647533Z Ran 1 test in 4.658s 2022-11-23T02:05:25.6647538Z 2022-11-23T02:05:25.6647588Z OK 2022-11-23T02:05:25.6647592Z 2022-11-23T02:05:25.6647668Z Generating XML reports... 2022-11-23T02:05:25.6648013Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014821.xml 2022-11-23T02:05:25.6648247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6648520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6648639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6648927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6649058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6649063Z 2022-11-23T02:05:25.6649128Z Running tests... 2022-11-23T02:05:25.6649372Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6649608Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17324 2022-11-23T02:05:25.6649761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17325 2022-11-23T02:05:25.6649949Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6650226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6650345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6650629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6650759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6650933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6651209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6651327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6651610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6651740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6651908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6652213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6652512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6652674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6652832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6652936Z skip: Skipped due to small world size. (3.742s) 2022-11-23T02:05:25.6652942Z 2022-11-23T02:05:25.6653140Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6653209Z Ran 1 test in 3.742s 2022-11-23T02:05:25.6653214Z 2022-11-23T02:05:25.6653277Z OK (skipped=1) 2022-11-23T02:05:25.6653282Z 2022-11-23T02:05:25.6653359Z Generating XML reports... 2022-11-23T02:05:25.6653701Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014828.xml 2022-11-23T02:05:25.6653971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6654252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6654369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6654652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6654779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6654784Z 2022-11-23T02:05:25.6654850Z Running tests... 2022-11-23T02:05:25.6655046Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6655268Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17507 2022-11-23T02:05:25.6655423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17508 2022-11-23T02:05:25.6655618Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6655928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6656046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6656325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6656455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6656627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6656901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6657019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6657302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6657432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6657604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6657909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6658201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6658364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6658520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6658628Z skip: Skipped due to small world size. (3.744s) 2022-11-23T02:05:25.6658633Z 2022-11-23T02:05:25.6658830Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6658898Z Ran 1 test in 3.744s 2022-11-23T02:05:25.6658903Z 2022-11-23T02:05:25.6658970Z OK (skipped=1) 2022-11-23T02:05:25.6658976Z 2022-11-23T02:05:25.6659053Z Generating XML reports... 2022-11-23T02:05:25.6659394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014835.xml 2022-11-23T02:05:25.6659625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6659901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6660018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6660301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6660460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6660469Z 2022-11-23T02:05:25.6660530Z Running tests... 2022-11-23T02:05:25.6660727Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6660957Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17690 2022-11-23T02:05:25.6661107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17691 2022-11-23T02:05:25.6661295Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6661571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6661687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6661971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6662099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6662301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6662579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6662697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6662979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6663107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6663279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6663583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6663884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6664042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6664199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6664303Z skip: Skipped due to small world size. (3.639s) 2022-11-23T02:05:25.6664308Z 2022-11-23T02:05:25.6664504Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6664572Z Ran 1 test in 3.639s 2022-11-23T02:05:25.6664577Z 2022-11-23T02:05:25.6664635Z OK (skipped=1) 2022-11-23T02:05:25.6664644Z 2022-11-23T02:05:25.6664716Z Generating XML reports... 2022-11-23T02:05:25.6665059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014842.xml 2022-11-23T02:05:25.6665295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6665574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6665693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6665975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6666102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6666106Z 2022-11-23T02:05:25.6666170Z Running tests... 2022-11-23T02:05:25.6666363Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6666601Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17873 2022-11-23T02:05:25.6666788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17874 2022-11-23T02:05:25.6666978Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6667257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6667375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6667657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6667785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6667957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6668232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6668352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6668635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6668761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6668960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6669262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6669561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6669721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6669880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6670056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6670221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6670522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6670966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6671028Z ok (4.940s) 2022-11-23T02:05:25.6671033Z 2022-11-23T02:05:25.6671230Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6671300Z Ran 1 test in 4.940s 2022-11-23T02:05:25.6671305Z 2022-11-23T02:05:25.6671359Z OK 2022-11-23T02:05:25.6671363Z 2022-11-23T02:05:25.6671440Z Generating XML reports... 2022-11-23T02:05:25.6671781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014849.xml 2022-11-23T02:05:25.6672018Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6672293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6672413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6672698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6672827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6672833Z 2022-11-23T02:05:25.6672897Z Running tests... 2022-11-23T02:05:25.6673094Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6673528Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s) 2022-11-23T02:05:25.6673569Z 2022-11-23T02:05:25.6673770Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6673839Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6673844Z 2022-11-23T02:05:25.6673903Z OK (skipped=1) 2022-11-23T02:05:25.6673912Z 2022-11-23T02:05:25.6673986Z Generating XML reports... 2022-11-23T02:05:25.6674325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014857.xml 2022-11-23T02:05:25.6674558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6674835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6674954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6675238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6675371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6675376Z 2022-11-23T02:05:25.6675440Z Running tests... 2022-11-23T02:05:25.6675634Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6675903Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18120 2022-11-23T02:05:25.6676106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18121 2022-11-23T02:05:25.6676300Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6676749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6676874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6677179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6677313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6677495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6677787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6677910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6678211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6678347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6678524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6678841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6679159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6679328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6679496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6679609Z skip: Skipped due to small world size. (3.837s) 2022-11-23T02:05:25.6679615Z 2022-11-23T02:05:25.6679826Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6679899Z Ran 1 test in 3.838s 2022-11-23T02:05:25.6679904Z 2022-11-23T02:05:25.6679971Z OK (skipped=1) 2022-11-23T02:05:25.6679976Z 2022-11-23T02:05:25.6680056Z Generating XML reports... 2022-11-23T02:05:25.6680417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014900.xml 2022-11-23T02:05:25.6680707Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6681000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6681127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6681427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6681563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6681568Z 2022-11-23T02:05:25.6681638Z Running tests... 2022-11-23T02:05:25.6681845Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6682088Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18303 2022-11-23T02:05:25.6682251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18304 2022-11-23T02:05:25.6682450Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6682778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6682905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6683199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6683334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6683517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6683836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6684131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6684254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6684556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6684693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6684876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6685193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6685361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6685529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6685595Z ok (3.740s) 2022-11-23T02:05:25.6685600Z 2022-11-23T02:05:25.6685809Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6685881Z Ran 1 test in 3.741s 2022-11-23T02:05:25.6685886Z 2022-11-23T02:05:25.6685942Z OK 2022-11-23T02:05:25.6685946Z 2022-11-23T02:05:25.6686027Z Generating XML reports... 2022-11-23T02:05:25.6686394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014906.xml 2022-11-23T02:05:25.6686639Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6687028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6687145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6687427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6687587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6687596Z 2022-11-23T02:05:25.6687656Z Running tests... 2022-11-23T02:05:25.6687852Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6688092Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18486 2022-11-23T02:05:25.6688245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18487 2022-11-23T02:05:25.6688434Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6688707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6688824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6689106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6689236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6689409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6689744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6690021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6690137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6690419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6690548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6690725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6691026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6691187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6691347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6691407Z ok (3.952s) 2022-11-23T02:05:25.6691413Z 2022-11-23T02:05:25.6691608Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6691677Z Ran 1 test in 3.953s 2022-11-23T02:05:25.6691683Z 2022-11-23T02:05:25.6691731Z OK 2022-11-23T02:05:25.6691739Z 2022-11-23T02:05:25.6691811Z Generating XML reports... 2022-11-23T02:05:25.6692155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014913.xml 2022-11-23T02:05:25.6692392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6692671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6692788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6693072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6693201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6693206Z 2022-11-23T02:05:25.6693270Z Running tests... 2022-11-23T02:05:25.6693467Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6693666Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:05:25.6693671Z 2022-11-23T02:05:25.6693865Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6693968Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6693973Z 2022-11-23T02:05:25.6694037Z OK (skipped=1) 2022-11-23T02:05:25.6694042Z 2022-11-23T02:05:25.6694119Z Generating XML reports... 2022-11-23T02:05:25.6694460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014920.xml 2022-11-23T02:05:25.6694698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6694974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6695092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6695376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6695504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6695509Z 2022-11-23T02:05:25.6695576Z Running tests... 2022-11-23T02:05:25.6695765Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6695948Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:05:25.6695957Z 2022-11-23T02:05:25.6696179Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6696249Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6696254Z 2022-11-23T02:05:25.6696316Z OK (skipped=1) 2022-11-23T02:05:25.6696321Z 2022-11-23T02:05:25.6696398Z Generating XML reports... 2022-11-23T02:05:25.6696736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014923.xml 2022-11-23T02:05:25.6696968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6697246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6697366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6697652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6697782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6697787Z 2022-11-23T02:05:25.6697855Z Running tests... 2022-11-23T02:05:25.6698052Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6698248Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:05:25.6698253Z 2022-11-23T02:05:25.6698449Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6698516Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6698521Z 2022-11-23T02:05:25.6698583Z OK (skipped=1) 2022-11-23T02:05:25.6698588Z 2022-11-23T02:05:25.6698664Z Generating XML reports... 2022-11-23T02:05:25.6699006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014926.xml 2022-11-23T02:05:25.6699238Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6699515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6699634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6699914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6700041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6700046Z 2022-11-23T02:05:25.6700110Z Running tests... 2022-11-23T02:05:25.6700305Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6700486Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:05:25.6700535Z 2022-11-23T02:05:25.6700728Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6700796Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6700801Z 2022-11-23T02:05:25.6700863Z OK (skipped=1) 2022-11-23T02:05:25.6700868Z 2022-11-23T02:05:25.6700944Z Generating XML reports... 2022-11-23T02:05:25.6701280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014929.xml 2022-11-23T02:05:25.6701512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6701787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6701903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6702183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6702314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6702319Z 2022-11-23T02:05:25.6702383Z Running tests... 2022-11-23T02:05:25.6702576Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6702796Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:05:25.6702802Z 2022-11-23T02:05:25.6702997Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6703065Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6703070Z 2022-11-23T02:05:25.6703134Z OK (skipped=1) 2022-11-23T02:05:25.6703139Z 2022-11-23T02:05:25.6703216Z Generating XML reports... 2022-11-23T02:05:25.6703555Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014932.xml 2022-11-23T02:05:25.6703781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6704061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6704178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6704466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6704596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6704601Z 2022-11-23T02:05:25.6704664Z Running tests... 2022-11-23T02:05:25.6704858Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6705051Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:05:25.6705056Z 2022-11-23T02:05:25.6705246Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6705313Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6705320Z 2022-11-23T02:05:25.6705383Z OK (skipped=1) 2022-11-23T02:05:25.6705388Z 2022-11-23T02:05:25.6705465Z Generating XML reports... 2022-11-23T02:05:25.6705803Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014935.xml 2022-11-23T02:05:25.6706038Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6706314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6706432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6706718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6706847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6706852Z 2022-11-23T02:05:25.6706916Z Running tests... 2022-11-23T02:05:25.6707143Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6707331Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:05:25.6707337Z 2022-11-23T02:05:25.6707528Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6707598Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6707603Z 2022-11-23T02:05:25.6707663Z OK (skipped=1) 2022-11-23T02:05:25.6707672Z 2022-11-23T02:05:25.6707744Z Generating XML reports... 2022-11-23T02:05:25.6708083Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014938.xml 2022-11-23T02:05:25.6708314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6708590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6708707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6708992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6709119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6709124Z 2022-11-23T02:05:25.6709188Z Running tests... 2022-11-23T02:05:25.6709413Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6709603Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:05:25.6709608Z 2022-11-23T02:05:25.6709802Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6709870Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6709875Z 2022-11-23T02:05:25.6709938Z OK (skipped=1) 2022-11-23T02:05:25.6709943Z 2022-11-23T02:05:25.6710019Z Generating XML reports... 2022-11-23T02:05:25.6710358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014941.xml 2022-11-23T02:05:25.6710593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6710869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6710990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6711275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6711403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6711407Z 2022-11-23T02:05:25.6711472Z Running tests... 2022-11-23T02:05:25.6711661Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6711881Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19133 2022-11-23T02:05:25.6712034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19134 2022-11-23T02:05:25.6712224Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6712502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6712618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6712902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6713029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6713200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6713472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6713625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6713908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6714037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6714211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6714513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6714814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6714974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6715133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6715390Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6715636Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6716151Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6716158Z 2022-11-23T02:05:25.6716425Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6716688Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6716935Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6717177Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6717431Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6717683Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6717934Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6718193Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6718436Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6718675Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6718922Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6719184Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6719441Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6719701Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6719953Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6720196Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6720446Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6720703Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6720952Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6721207Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6721485Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6721724Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6721975Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6722231Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6722480Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6722736Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6722977Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6723219Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6723468Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6723750Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6724000Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6724255Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6724497Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6724733Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6724981Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6725239Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6725487Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6725745Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6725990Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6726234Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6726482Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6726739Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6726984Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6727242Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6727484Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6727728Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6727973Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6728230Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6728480Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6728737Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6728977Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6729253Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6729498Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6729751Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6729997Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6730251Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6730495Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6730735Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6730985Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6731241Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6731488Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6731775Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6732017Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6732260Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6732503Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6732759Z STAGE:2022-11-23 01:49:46 19134:19134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6733008Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6733264Z STAGE:2022-11-23 01:49:46 19133:19133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6733325Z ok (3.940s) 2022-11-23T02:05:25.6733330Z 2022-11-23T02:05:25.6733530Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6733598Z Ran 1 test in 3.941s 2022-11-23T02:05:25.6733603Z 2022-11-23T02:05:25.6733656Z OK 2022-11-23T02:05:25.6733660Z 2022-11-23T02:05:25.6733739Z Generating XML reports... 2022-11-23T02:05:25.6734082Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014944.xml 2022-11-23T02:05:25.6734318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6734596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6734713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6734998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6735130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6735135Z 2022-11-23T02:05:25.6735199Z Running tests... 2022-11-23T02:05:25.6735396Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6736089Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.511s) 2022-11-23T02:05:25.6736095Z 2022-11-23T02:05:25.6736287Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6736393Z Ran 1 test in 0.511s 2022-11-23T02:05:25.6736398Z 2022-11-23T02:05:25.6736461Z OK (skipped=1) 2022-11-23T02:05:25.6736465Z 2022-11-23T02:05:25.6736542Z Generating XML reports... 2022-11-23T02:05:25.6736888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014951.xml 2022-11-23T02:05:25.6737122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6737399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6737517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6737799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6737929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6737936Z 2022-11-23T02:05:25.6738001Z Running tests... 2022-11-23T02:05:25.6738196Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6738425Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19380 2022-11-23T02:05:25.6738611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19381 2022-11-23T02:05:25.6738804Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6739081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6739202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6739487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6739621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6739795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6740067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6740188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6740473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6740600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6740770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6741073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6741373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6741534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6741691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6741863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6742029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6742327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6742623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6742877Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6743156Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6743411Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6743661Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6743930Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6744193Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6744441Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6744681Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6744929Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6745191Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6745441Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6745810Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6746057Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6746295Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6746541Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6746797Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6747044Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6747304Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6747546Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6747785Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6748031Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6748285Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6748529Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6748786Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6749033Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6749282Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6749528Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6749786Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6750033Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6750291Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6750536Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6750775Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6751057Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6751312Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6751563Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6751814Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6752054Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6752292Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6752536Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6752792Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6753037Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6753290Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6753576Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6753817Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6754060Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6754315Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6754562Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6754815Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6755057Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6755297Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6755545Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6755801Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6756097Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6756359Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6756605Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6756847Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6757100Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6757360Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6757615Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6757875Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6758111Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6758349Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6758597Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6758893Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6759144Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6759405Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6759648Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6759886Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.6760133Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6760388Z STAGE:2022-11-23 01:49:57 19381:19381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6760637Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.6760895Z STAGE:2022-11-23 01:49:57 19380:19380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.6760957Z ok (4.039s) 2022-11-23T02:05:25.6760963Z 2022-11-23T02:05:25.6761160Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6761270Z Ran 1 test in 4.039s 2022-11-23T02:05:25.6761276Z 2022-11-23T02:05:25.6761330Z OK 2022-11-23T02:05:25.6761335Z 2022-11-23T02:05:25.6761413Z Generating XML reports... 2022-11-23T02:05:25.6761760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014954.xml 2022-11-23T02:05:25.6761995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6762274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6762392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6762678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6762803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6762812Z 2022-11-23T02:05:25.6762872Z Running tests... 2022-11-23T02:05:25.6763075Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6763305Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19575 2022-11-23T02:05:25.6763456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19576 2022-11-23T02:05:25.6763647Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6763923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6764043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6764329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6764457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6764632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6764909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6765025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6765309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6765438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6765611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6765963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6766265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6766427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6766585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6766689Z skip: Skipped due to small world size. (3.656s) 2022-11-23T02:05:25.6766695Z 2022-11-23T02:05:25.6766892Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6766961Z Ran 1 test in 3.657s 2022-11-23T02:05:25.6766966Z 2022-11-23T02:05:25.6767025Z OK (skipped=1) 2022-11-23T02:05:25.6767035Z 2022-11-23T02:05:25.6767108Z Generating XML reports... 2022-11-23T02:05:25.6767454Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015001.xml 2022-11-23T02:05:25.6767685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6767996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6768114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6768398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6768526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6768531Z 2022-11-23T02:05:25.6768595Z Running tests... 2022-11-23T02:05:25.6768788Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6769019Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19758 2022-11-23T02:05:25.6769172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19759 2022-11-23T02:05:25.6769363Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6769640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6769758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6770040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6770168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6770339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6770615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6770734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6771019Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6771149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6771320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6771618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6771916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6772076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6772232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6772866Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:05:25.6772935Z warnings.warn( 2022-11-23T02:05:25.6773522Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:05:25.6773588Z warnings.warn( 2022-11-23T02:05:25.6773648Z ok (4.238s) 2022-11-23T02:05:25.6773653Z 2022-11-23T02:05:25.6773850Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6773919Z Ran 1 test in 4.239s 2022-11-23T02:05:25.6773926Z 2022-11-23T02:05:25.6773979Z OK 2022-11-23T02:05:25.6773983Z 2022-11-23T02:05:25.6774060Z Generating XML reports... 2022-11-23T02:05:25.6774400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015008.xml 2022-11-23T02:05:25.6774667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6774947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6775066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6775349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6775481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6775487Z 2022-11-23T02:05:25.6775551Z Running tests... 2022-11-23T02:05:25.6775753Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6776451Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:05:25.6776457Z 2022-11-23T02:05:25.6776653Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6776721Z Ran 1 test in 0.509s 2022-11-23T02:05:25.6776726Z 2022-11-23T02:05:25.6776790Z OK (skipped=1) 2022-11-23T02:05:25.6776794Z 2022-11-23T02:05:25.6776871Z Generating XML reports... 2022-11-23T02:05:25.6777209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015015.xml 2022-11-23T02:05:25.6777435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6777713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6777832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6778118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6778246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6778251Z 2022-11-23T02:05:25.6778315Z Running tests... 2022-11-23T02:05:25.6778508Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6778896Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:05:25.6778901Z 2022-11-23T02:05:25.6779091Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6779195Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6779200Z 2022-11-23T02:05:25.6779264Z OK (skipped=1) 2022-11-23T02:05:25.6779268Z 2022-11-23T02:05:25.6779345Z Generating XML reports... 2022-11-23T02:05:25.6779688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015019.xml 2022-11-23T02:05:25.6779920Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6780195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6780312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6780593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6780721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6780728Z 2022-11-23T02:05:25.6780793Z Running tests... 2022-11-23T02:05:25.6780985Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6781408Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:05:25.6781415Z 2022-11-23T02:05:25.6781610Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6781679Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6781684Z 2022-11-23T02:05:25.6781749Z OK (skipped=1) 2022-11-23T02:05:25.6781754Z 2022-11-23T02:05:25.6781830Z Generating XML reports... 2022-11-23T02:05:25.6782165Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015022.xml 2022-11-23T02:05:25.6782397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6782677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6782794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6783078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6783210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6783215Z 2022-11-23T02:05:25.6783280Z Running tests... 2022-11-23T02:05:25.6783476Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6783709Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20115 2022-11-23T02:05:25.6783859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20116 2022-11-23T02:05:25.6784049Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6784328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6784445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6784730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6784858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6785029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6785333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6785607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6785726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6786052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6786180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6786356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6786657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6786810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6786969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6787151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmkoqo0qg 2022-11-23T02:05:25.6787340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmkoqo0qg/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6787519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo5776ar 2022-11-23T02:05:25.6787704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo5776ar/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6787899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6788065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6788127Z ok (6.257s) 2022-11-23T02:05:25.6788132Z 2022-11-23T02:05:25.6788332Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6788400Z Ran 1 test in 6.258s 2022-11-23T02:05:25.6788405Z 2022-11-23T02:05:25.6788457Z OK 2022-11-23T02:05:25.6788461Z 2022-11-23T02:05:25.6788537Z Generating XML reports... 2022-11-23T02:05:25.6788882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015025.xml 2022-11-23T02:05:25.6789118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6789395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6789517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6789800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6789928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6789932Z 2022-11-23T02:05:25.6789996Z Running tests... 2022-11-23T02:05:25.6790191Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6790426Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20308 2022-11-23T02:05:25.6790582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20309 2022-11-23T02:05:25.6790772Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6791052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6791171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6791455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6791584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6791757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6792059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6792372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6792490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6792774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6792907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6793080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6793385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6793546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6793703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6793887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsewj5k1 2022-11-23T02:05:25.6794077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsewj5k1/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6794257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnpcd798 2022-11-23T02:05:25.6794477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnpcd798/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6794642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6794806Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6794959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6795121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6795183Z ok (6.357s) 2022-11-23T02:05:25.6795188Z 2022-11-23T02:05:25.6795391Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6795460Z Ran 1 test in 6.357s 2022-11-23T02:05:25.6795465Z 2022-11-23T02:05:25.6795521Z OK 2022-11-23T02:05:25.6795525Z 2022-11-23T02:05:25.6795603Z Generating XML reports... 2022-11-23T02:05:25.6795952Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015034.xml 2022-11-23T02:05:25.6796244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6796523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6796640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6796925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6797054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6797062Z 2022-11-23T02:05:25.6797127Z Running tests... 2022-11-23T02:05:25.6797323Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6798030Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:05:25.6798036Z 2022-11-23T02:05:25.6798232Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6798300Z Ran 1 test in 0.510s 2022-11-23T02:05:25.6798305Z 2022-11-23T02:05:25.6798367Z OK (skipped=1) 2022-11-23T02:05:25.6798371Z 2022-11-23T02:05:25.6798448Z Generating XML reports... 2022-11-23T02:05:25.6798787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015043.xml 2022-11-23T02:05:25.6799061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6799336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6799456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6799734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6799863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6799873Z 2022-11-23T02:05:25.6799933Z Running tests... 2022-11-23T02:05:25.6800135Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6800855Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.519s) 2022-11-23T02:05:25.6800867Z 2022-11-23T02:05:25.6801093Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6801162Z Ran 1 test in 0.519s 2022-11-23T02:05:25.6801167Z 2022-11-23T02:05:25.6801235Z OK (skipped=1) 2022-11-23T02:05:25.6801239Z 2022-11-23T02:05:25.6801316Z Generating XML reports... 2022-11-23T02:05:25.6801659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015047.xml 2022-11-23T02:05:25.6801893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6802168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6802289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6802578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6802708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6802712Z 2022-11-23T02:05:25.6802779Z Running tests... 2022-11-23T02:05:25.6802972Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6803325Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.003s) 2022-11-23T02:05:25.6803330Z 2022-11-23T02:05:25.6803523Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6803591Z Ran 1 test in 0.003s 2022-11-23T02:05:25.6803595Z 2022-11-23T02:05:25.6803657Z OK (skipped=1) 2022-11-23T02:05:25.6803662Z 2022-11-23T02:05:25.6803740Z Generating XML reports... 2022-11-23T02:05:25.6804081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015050.xml 2022-11-23T02:05:25.6804311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6804590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6804706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6804988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6805112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6805121Z 2022-11-23T02:05:25.6805181Z Running tests... 2022-11-23T02:05:25.6805374Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6805634Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20675 2022-11-23T02:05:25.6805834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20676 2022-11-23T02:05:25.6806026Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6806304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6806420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6806705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6806834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6807005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6807310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6807583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6807733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6808017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6808144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6808315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6808616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6808777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6808938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6809121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwufb0d_ 2022-11-23T02:05:25.6809315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwufb0d_/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6809493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyl9du28 2022-11-23T02:05:25.6809674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyl9du28/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6809734Z ok (4.156s) 2022-11-23T02:05:25.6809739Z 2022-11-23T02:05:25.6809936Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6810003Z Ran 1 test in 4.157s 2022-11-23T02:05:25.6810008Z 2022-11-23T02:05:25.6810061Z OK 2022-11-23T02:05:25.6810065Z 2022-11-23T02:05:25.6810143Z Generating XML reports... 2022-11-23T02:05:25.6810486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015053.xml 2022-11-23T02:05:25.6810718Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6810995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6811112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6811396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6811524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6811529Z 2022-11-23T02:05:25.6811594Z Running tests... 2022-11-23T02:05:25.6811789Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6812019Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20858 2022-11-23T02:05:25.6812205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20859 2022-11-23T02:05:25.6812394Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6812676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6812792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6813074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6813202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6813373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6813645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6813764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6814047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6814207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6814380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6814688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6814985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6815143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6815302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6815487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjd19u5dw 2022-11-23T02:05:25.6815676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjd19u5dw/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6815852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccgedhkv 2022-11-23T02:05:25.6816040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccgedhkv/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6816203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6816366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6816426Z ok (6.248s) 2022-11-23T02:05:25.6816431Z 2022-11-23T02:05:25.6816629Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6816697Z Ran 1 test in 6.248s 2022-11-23T02:05:25.6816704Z 2022-11-23T02:05:25.6816758Z OK 2022-11-23T02:05:25.6816762Z 2022-11-23T02:05:25.6816840Z Generating XML reports... 2022-11-23T02:05:25.6817181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015100.xml 2022-11-23T02:05:25.6817414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6817686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6817803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6818088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6818218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6818223Z 2022-11-23T02:05:25.6818289Z Running tests... 2022-11-23T02:05:25.6818520Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6818876Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.004s) 2022-11-23T02:05:25.6818881Z 2022-11-23T02:05:25.6819077Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6819145Z Ran 1 test in 0.004s 2022-11-23T02:05:25.6819149Z 2022-11-23T02:05:25.6819214Z OK (skipped=1) 2022-11-23T02:05:25.6819219Z 2022-11-23T02:05:25.6819295Z Generating XML reports... 2022-11-23T02:05:25.6819635Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015110.xml 2022-11-23T02:05:25.6819868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6820145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6820264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6820548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6820677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6820682Z 2022-11-23T02:05:25.6820778Z Running tests... 2022-11-23T02:05:25.6820974Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6821322Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.003s) 2022-11-23T02:05:25.6821327Z 2022-11-23T02:05:25.6821517Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6821585Z Ran 1 test in 0.003s 2022-11-23T02:05:25.6821589Z 2022-11-23T02:05:25.6821651Z OK (skipped=1) 2022-11-23T02:05:25.6821656Z 2022-11-23T02:05:25.6821735Z Generating XML reports... 2022-11-23T02:05:25.6822070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015113.xml 2022-11-23T02:05:25.6822304Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6822579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6822696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6822978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6823107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6823112Z 2022-11-23T02:05:25.6823177Z Running tests... 2022-11-23T02:05:25.6823372Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6823602Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21167 2022-11-23T02:05:25.6823757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21168 2022-11-23T02:05:25.6823947Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6824223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6824344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6824628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6824754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6824927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6825204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6825361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6825646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6825776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6825951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6826256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6826549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6826707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6826888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88dcjirk 2022-11-23T02:05:25.6827079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88dcjirk/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6827238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6827447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1axilwnx 2022-11-23T02:05:25.6827638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1axilwnx/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6828342Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6829033Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6829966Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1122.) 2022-11-23T02:05:25.6830132Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T02:05:25.6830294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6831201Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1122.) 2022-11-23T02:05:25.6831359Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T02:05:25.6831523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6832208Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6832931Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6833610Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6834321Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6835000Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6835671Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6836642Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6837362Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6838074Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6838779Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:05:25.6838845Z ok (3.930s) 2022-11-23T02:05:25.6838850Z 2022-11-23T02:05:25.6839061Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6839134Z Ran 1 test in 3.931s 2022-11-23T02:05:25.6839186Z 2022-11-23T02:05:25.6839242Z OK 2022-11-23T02:05:25.6839247Z 2022-11-23T02:05:25.6839328Z Generating XML reports... 2022-11-23T02:05:25.6839694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015115.xml 2022-11-23T02:05:25.6839945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6840241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6840366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6840668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6840804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6840810Z 2022-11-23T02:05:25.6840879Z Running tests... 2022-11-23T02:05:25.6841091Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6841421Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.004s) 2022-11-23T02:05:25.6841426Z 2022-11-23T02:05:25.6841635Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6841743Z Ran 1 test in 0.004s 2022-11-23T02:05:25.6841750Z 2022-11-23T02:05:25.6841818Z OK (skipped=1) 2022-11-23T02:05:25.6841822Z 2022-11-23T02:05:25.6841905Z Generating XML reports... 2022-11-23T02:05:25.6842266Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015122.xml 2022-11-23T02:05:25.6842517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6842805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6842934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6843239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6843379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6843384Z 2022-11-23T02:05:25.6843456Z Running tests... 2022-11-23T02:05:25.6843666Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6844020Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:05:25.6844025Z 2022-11-23T02:05:25.6844229Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6844301Z Ran 1 test in 0.003s 2022-11-23T02:05:25.6844307Z 2022-11-23T02:05:25.6844374Z OK (skipped=1) 2022-11-23T02:05:25.6844378Z 2022-11-23T02:05:25.6844461Z Generating XML reports... 2022-11-23T02:05:25.6844821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015125.xml 2022-11-23T02:05:25.6845066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6845359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6845483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6845782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6845923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6845928Z 2022-11-23T02:05:25.6845995Z Running tests... 2022-11-23T02:05:25.6846202Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6846942Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.568s) 2022-11-23T02:05:25.6846981Z 2022-11-23T02:05:25.6847192Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6847264Z Ran 1 test in 0.568s 2022-11-23T02:05:25.6847269Z 2022-11-23T02:05:25.6847336Z OK (skipped=1) 2022-11-23T02:05:25.6847340Z 2022-11-23T02:05:25.6847422Z Generating XML reports... 2022-11-23T02:05:25.6847894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015128.xml 2022-11-23T02:05:25.6848128Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6848403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6848525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6848809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6848971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6848976Z 2022-11-23T02:05:25.6849042Z Running tests... 2022-11-23T02:05:25.6849238Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6849934Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.493s) 2022-11-23T02:05:25.6849940Z 2022-11-23T02:05:25.6850135Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6850204Z Ran 1 test in 0.493s 2022-11-23T02:05:25.6850209Z 2022-11-23T02:05:25.6850274Z OK (skipped=1) 2022-11-23T02:05:25.6850279Z 2022-11-23T02:05:25.6850355Z Generating XML reports... 2022-11-23T02:05:25.6850695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015132.xml 2022-11-23T02:05:25.6850927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6851206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6851322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6851603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6851731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6851739Z 2022-11-23T02:05:25.6851804Z Running tests... 2022-11-23T02:05:25.6852002Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6852256Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21586 2022-11-23T02:05:25.6852411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21587 2022-11-23T02:05:25.6852602Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6852878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6852995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6853279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6853439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6853608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6853888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6854006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6854293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6854421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6854592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6854893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6855196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6855355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6855555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6855725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.6855892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.6856194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6856492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.6856674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppq_gt2dh 2022-11-23T02:05:25.6856867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppq_gt2dh/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6857044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y7un66y 2022-11-23T02:05:25.6857233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y7un66y/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6857399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6857566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6857727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6857889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6857948Z ok (6.950s) 2022-11-23T02:05:25.6857953Z 2022-11-23T02:05:25.6858148Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6858219Z Ran 1 test in 6.951s 2022-11-23T02:05:25.6858224Z 2022-11-23T02:05:25.6858279Z OK 2022-11-23T02:05:25.6858283Z 2022-11-23T02:05:25.6858360Z Generating XML reports... 2022-11-23T02:05:25.6858704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015135.xml 2022-11-23T02:05:25.6858937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6859215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6859334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6859621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6859753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6859791Z 2022-11-23T02:05:25.6859859Z Running tests... 2022-11-23T02:05:25.6860056Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6860298Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21785 2022-11-23T02:05:25.6860451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21786 2022-11-23T02:05:25.6860641Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6860915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6861032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6861316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6861452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6861623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6861927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6862233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6862347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6862633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6862762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6862931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6863231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6863393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6863592Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:05:25.6863748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6863937Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:05:25.6864120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4yvjfht 2022-11-23T02:05:25.6864310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4yvjfht/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6864486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1nril86 2022-11-23T02:05:25.6864676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1nril86/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6864840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6865001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6865164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6865325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6865523Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:05:25.6865721Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:05:25.6865914Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:05:25.6866136Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:05:25.6866295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6866457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6866613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6866774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6866965Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:05:25.6867158Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:05:25.6867351Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:05:25.6867542Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:05:25.6867699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6867858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6868045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6868207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6868267Z ok (7.450s) 2022-11-23T02:05:25.6868273Z 2022-11-23T02:05:25.6868472Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6868541Z Ran 1 test in 7.450s 2022-11-23T02:05:25.6868546Z 2022-11-23T02:05:25.6868600Z OK 2022-11-23T02:05:25.6868604Z 2022-11-23T02:05:25.6868680Z Generating XML reports... 2022-11-23T02:05:25.6869028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015145.xml 2022-11-23T02:05:25.6869272Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6869551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6869669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6869956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6870084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6870089Z 2022-11-23T02:05:25.6870153Z Running tests... 2022-11-23T02:05:25.6870344Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6871044Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:05:25.6871053Z 2022-11-23T02:05:25.6871250Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6871314Z Ran 1 test in 0.509s 2022-11-23T02:05:25.6871324Z 2022-11-23T02:05:25.6871383Z OK (skipped=1) 2022-11-23T02:05:25.6871388Z 2022-11-23T02:05:25.6871465Z Generating XML reports... 2022-11-23T02:05:25.6871806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015156.xml 2022-11-23T02:05:25.6872039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6872317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6872469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6872755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6872884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6872892Z 2022-11-23T02:05:25.6872956Z Running tests... 2022-11-23T02:05:25.6873152Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6873392Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22036 2022-11-23T02:05:25.6873545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22037 2022-11-23T02:05:25.6873735Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6874011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6874131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6874416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6874578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6874753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6875032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6875151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6875434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6879576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6879781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6880117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6880421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6880583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6880991Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:05:25.6881151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6881555Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:05:25.6881738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprlgvsno 2022-11-23T02:05:25.6881930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprlgvsno/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6882102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1g464j4 2022-11-23T02:05:25.6882285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1g464j4/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6882447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6882697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6882892Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:05:25.6883088Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:05:25.6883298Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:05:25.6883508Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:05:25.6883741Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:05:25.6883972Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:05:25.6884204Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:05:25.6884472Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:05:25.6884635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6884796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6884856Z ok (6.642s) 2022-11-23T02:05:25.6884863Z 2022-11-23T02:05:25.6885067Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6885136Z Ran 1 test in 6.643s 2022-11-23T02:05:25.6885141Z 2022-11-23T02:05:25.6885194Z OK 2022-11-23T02:05:25.6885198Z 2022-11-23T02:05:25.6885274Z Generating XML reports... 2022-11-23T02:05:25.6885622Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015159.xml 2022-11-23T02:05:25.6885856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6886139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6886259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6886547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6886676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6886681Z 2022-11-23T02:05:25.6886746Z Running tests... 2022-11-23T02:05:25.6886939Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6887209Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22229 2022-11-23T02:05:25.6887363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22230 2022-11-23T02:05:25.6887555Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6887834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6887952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6888232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6888361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6888534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6888851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6888968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6889255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6889382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6889552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6889854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6890157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6890320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6890480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6890674Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6890885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbqv_zdy 2022-11-23T02:05:25.6891076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbqv_zdy/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6891265Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6891444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrvqzsn0 2022-11-23T02:05:25.6891628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrvqzsn0/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6891796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6891963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6892127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6892289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6892351Z ok (7.395s) 2022-11-23T02:05:25.6892357Z 2022-11-23T02:05:25.6892559Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6892628Z Ran 1 test in 7.396s 2022-11-23T02:05:25.6892634Z 2022-11-23T02:05:25.6892686Z OK 2022-11-23T02:05:25.6892691Z 2022-11-23T02:05:25.6892767Z Generating XML reports... 2022-11-23T02:05:25.6893114Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015209.xml 2022-11-23T02:05:25.6893348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6893630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6893749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6894037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6894166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6894171Z 2022-11-23T02:05:25.6894236Z Running tests... 2022-11-23T02:05:25.6894432Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6894695Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22476 2022-11-23T02:05:25.6894847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22477 2022-11-23T02:05:25.6895035Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6895347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6895463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6895753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6895883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6896054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6896332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6896448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6896732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6896861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6897032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6897384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6897686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6897847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6898005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6898195Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6898384Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6898565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb98q7fqw 2022-11-23T02:05:25.6898759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb98q7fqw/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6898938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2vowmf83 2022-11-23T02:05:25.6899129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2vowmf83/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6899295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6899453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6899616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6899778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6899840Z ok (6.847s) 2022-11-23T02:05:25.6899846Z 2022-11-23T02:05:25.6900046Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6900116Z Ran 1 test in 6.847s 2022-11-23T02:05:25.6900122Z 2022-11-23T02:05:25.6900177Z OK 2022-11-23T02:05:25.6900182Z 2022-11-23T02:05:25.6900260Z Generating XML reports... 2022-11-23T02:05:25.6900603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015220.xml 2022-11-23T02:05:25.6900838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6901117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6901235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6901517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6901680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6901685Z 2022-11-23T02:05:25.6901749Z Running tests... 2022-11-23T02:05:25.6901952Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6902268Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22723 2022-11-23T02:05:25.6902419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22724 2022-11-23T02:05:25.6902608Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6902884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6903005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6903289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6903413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6903616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6903921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6904195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6904313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6904594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6904725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6904894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6905198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6905356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6905514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6905702Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6905883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvhcoen7 2022-11-23T02:05:25.6906076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvhcoen7/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6906261Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6906438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2mios4u 2022-11-23T02:05:25.6906628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2mios4u/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6906791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6906955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6907114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6907274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6907334Z ok (6.939s) 2022-11-23T02:05:25.6907339Z 2022-11-23T02:05:25.6907537Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6907651Z Ran 1 test in 6.940s 2022-11-23T02:05:25.6907660Z 2022-11-23T02:05:25.6907709Z OK 2022-11-23T02:05:25.6907713Z 2022-11-23T02:05:25.6907789Z Generating XML reports... 2022-11-23T02:05:25.6908138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015230.xml 2022-11-23T02:05:25.6908372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6908649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6908768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6909052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6909184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6909191Z 2022-11-23T02:05:25.6909257Z Running tests... 2022-11-23T02:05:25.6909456Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6909797Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22970 2022-11-23T02:05:25.6909950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22971 2022-11-23T02:05:25.6910141Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6910419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6910539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6910825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6910958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6911129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6911409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6911527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6911807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6911936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6912104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6912405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6912706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6912866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6913026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6913214Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6913394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphr5v5cy_ 2022-11-23T02:05:25.6913584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphr5v5cy_/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6913768Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6913942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqbohxpd 2022-11-23T02:05:25.6914163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqbohxpd/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6914327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6914491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6914652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6914810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6914871Z ok (7.148s) 2022-11-23T02:05:25.6914876Z 2022-11-23T02:05:25.6915076Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6915144Z Ran 1 test in 7.148s 2022-11-23T02:05:25.6915149Z 2022-11-23T02:05:25.6915201Z OK 2022-11-23T02:05:25.6915205Z 2022-11-23T02:05:25.6915283Z Generating XML reports... 2022-11-23T02:05:25.6915627Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015240.xml 2022-11-23T02:05:25.6915859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6916387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6916518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6916826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6916963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6916968Z 2022-11-23T02:05:25.6917035Z Running tests... 2022-11-23T02:05:25.6917245Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6917574Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23217 2022-11-23T02:05:25.6917738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23218 2022-11-23T02:05:25.6917940Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6918230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6918355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6918653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6918789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6918974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6919296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6919591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6919716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6920017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6920151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6920332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6920652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6920820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6921027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6921228Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6921425Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6921610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6t0jz0gn 2022-11-23T02:05:25.6921810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6t0jz0gn/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6921997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilbzqupf 2022-11-23T02:05:25.6922199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilbzqupf/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6922371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6922544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6922711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6922907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6922972Z ok (6.856s) 2022-11-23T02:05:25.6922977Z 2022-11-23T02:05:25.6923189Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6923261Z Ran 1 test in 6.857s 2022-11-23T02:05:25.6923267Z 2022-11-23T02:05:25.6923323Z OK 2022-11-23T02:05:25.6923328Z 2022-11-23T02:05:25.6923407Z Generating XML reports... 2022-11-23T02:05:25.6923773Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015250.xml 2022-11-23T02:05:25.6924021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6924316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6924440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6924745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6924881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6924887Z 2022-11-23T02:05:25.6924955Z Running tests... 2022-11-23T02:05:25.6925166Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6925486Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23464 2022-11-23T02:05:25.6925644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23465 2022-11-23T02:05:25.6925847Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6926136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6926265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6926566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6926700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6926882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6927269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6927387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6927703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6927831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6928004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6928305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6928599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6928757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6928915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6929104Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6929286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp406dp7rj 2022-11-23T02:05:25.6929476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp406dp7rj/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6929690Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6929866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2k380wnz 2022-11-23T02:05:25.6930049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2k380wnz/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6930211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6930370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6930531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6930695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6930755Z ok (6.859s) 2022-11-23T02:05:25.6930760Z 2022-11-23T02:05:25.6930960Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6931033Z Ran 1 test in 6.860s 2022-11-23T02:05:25.6931038Z 2022-11-23T02:05:25.6931091Z OK 2022-11-23T02:05:25.6931095Z 2022-11-23T02:05:25.6931171Z Generating XML reports... 2022-11-23T02:05:25.6931518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015300.xml 2022-11-23T02:05:25.6931751Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6932028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6932145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6932432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6932562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6932567Z 2022-11-23T02:05:25.6932631Z Running tests... 2022-11-23T02:05:25.6932830Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6933138Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23711 2022-11-23T02:05:25.6933291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23712 2022-11-23T02:05:25.6933480Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6933755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6933907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6934189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6934319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6934487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6934790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6935069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6935184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6935470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6935599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6935769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6936102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6936261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6936419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6936607Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6936786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw_gbjph 2022-11-23T02:05:25.6936975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw_gbjph/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6937159Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6937331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtzstgj6 2022-11-23T02:05:25.6937519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtzstgj6/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6937680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6937844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6938007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6938169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6938229Z ok (6.948s) 2022-11-23T02:05:25.6938236Z 2022-11-23T02:05:25.6938436Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6938499Z Ran 1 test in 6.949s 2022-11-23T02:05:25.6938510Z 2022-11-23T02:05:25.6938558Z OK 2022-11-23T02:05:25.6938562Z 2022-11-23T02:05:25.6938639Z Generating XML reports... 2022-11-23T02:05:25.6938988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015310.xml 2022-11-23T02:05:25.6939220Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6939498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6939616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6939904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6940032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6940069Z 2022-11-23T02:05:25.6940135Z Running tests... 2022-11-23T02:05:25.6940332Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6940646Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23958 2022-11-23T02:05:25.6940797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23959 2022-11-23T02:05:25.6940988Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6941265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6941384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6941667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6941798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6941971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6942281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6942400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6942685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6942815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6942979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6943281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6943589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6943749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6943907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6944098Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6944280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnwus2_z 2022-11-23T02:05:25.6944473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnwus2_z/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6944659Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6944838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptczrs8do 2022-11-23T02:05:25.6945023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptczrs8do/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6945189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6945351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6945514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6945673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6945733Z ok (7.159s) 2022-11-23T02:05:25.6945738Z 2022-11-23T02:05:25.6945938Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6946006Z Ran 1 test in 7.159s 2022-11-23T02:05:25.6946011Z 2022-11-23T02:05:25.6946064Z OK 2022-11-23T02:05:25.6946100Z 2022-11-23T02:05:25.6946177Z Generating XML reports... 2022-11-23T02:05:25.6946525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015320.xml 2022-11-23T02:05:25.6946762Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6947044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6947158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6947446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6947576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6947581Z 2022-11-23T02:05:25.6947646Z Running tests... 2022-11-23T02:05:25.6947841Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6948156Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24205 2022-11-23T02:05:25.6948307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24206 2022-11-23T02:05:25.6948526Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6948804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6948921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6949205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6949333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6949506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6949779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6949896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6950182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6950310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6950483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6950786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6951088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6951249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6951407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6951598Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6951780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupnkgfwg 2022-11-23T02:05:25.6951966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupnkgfwg/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6952150Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6952325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5m5oxvm 2022-11-23T02:05:25.6952512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5m5oxvm/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6952706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6952870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6953029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6953191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6953252Z ok (6.835s) 2022-11-23T02:05:25.6953257Z 2022-11-23T02:05:25.6953458Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6953526Z Ran 1 test in 6.835s 2022-11-23T02:05:25.6953531Z 2022-11-23T02:05:25.6953582Z OK 2022-11-23T02:05:25.6953587Z 2022-11-23T02:05:25.6953663Z Generating XML reports... 2022-11-23T02:05:25.6954009Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015330.xml 2022-11-23T02:05:25.6954243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6954520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6954636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6954967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6955098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6955103Z 2022-11-23T02:05:25.6955168Z Running tests... 2022-11-23T02:05:25.6955365Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6955670Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24452 2022-11-23T02:05:25.6955824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24453 2022-11-23T02:05:25.6956009Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6956354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6956474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6956758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6956888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6957059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6957337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6957455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6957737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6957866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6958036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6958342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6958641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6958799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6958955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6959266Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6959447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmwen6tv 2022-11-23T02:05:25.6959640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmwen6tv/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6959827Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6960004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpift6bt5g 2022-11-23T02:05:25.6960193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpift6bt5g/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6960355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6960517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6960676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6960837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6960897Z ok (7.147s) 2022-11-23T02:05:25.6960902Z 2022-11-23T02:05:25.6961138Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6961209Z Ran 1 test in 7.147s 2022-11-23T02:05:25.6961214Z 2022-11-23T02:05:25.6961266Z OK 2022-11-23T02:05:25.6961270Z 2022-11-23T02:05:25.6961346Z Generating XML reports... 2022-11-23T02:05:25.6961691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015340.xml 2022-11-23T02:05:25.6961922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6962199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6962320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6962607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6962735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6962742Z 2022-11-23T02:05:25.6962806Z Running tests... 2022-11-23T02:05:25.6963004Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6963274Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24699 2022-11-23T02:05:25.6963426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24700 2022-11-23T02:05:25.6963616Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6963897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6964015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6964300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6964428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6964596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6964901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6965175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6965293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6965614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6965743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6965917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6966220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6966379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6966537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6966724Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6966904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3e31rrqh 2022-11-23T02:05:25.6967095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3e31rrqh/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6967278Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6967485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5nxq1kk 2022-11-23T02:05:25.6967672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5nxq1kk/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6967837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6968001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6968161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6968322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6968383Z ok (6.959s) 2022-11-23T02:05:25.6968388Z 2022-11-23T02:05:25.6968587Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6968650Z Ran 1 test in 6.959s 2022-11-23T02:05:25.6968660Z 2022-11-23T02:05:25.6968707Z OK 2022-11-23T02:05:25.6968712Z 2022-11-23T02:05:25.6968788Z Generating XML reports... 2022-11-23T02:05:25.6969134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015350.xml 2022-11-23T02:05:25.6969369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6969648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6969764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6970048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6970179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6970184Z 2022-11-23T02:05:25.6970249Z Running tests... 2022-11-23T02:05:25.6970444Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6970711Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24946 2022-11-23T02:05:25.6970864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24947 2022-11-23T02:05:25.6971051Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6971325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6971442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6971728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6971891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6972066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6972373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6972652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6972769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6973055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6973179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6973349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6973652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6973810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6974001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6974191Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6974373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfdiityp 2022-11-23T02:05:25.6974564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfdiityp/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6974749Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6974924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bnglzuv 2022-11-23T02:05:25.6975109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bnglzuv/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6975271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6975429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6975590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6975750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6975812Z ok (6.846s) 2022-11-23T02:05:25.6975817Z 2022-11-23T02:05:25.6976017Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6976084Z Ran 1 test in 6.846s 2022-11-23T02:05:25.6976090Z 2022-11-23T02:05:25.6976143Z OK 2022-11-23T02:05:25.6976150Z 2022-11-23T02:05:25.6976227Z Generating XML reports... 2022-11-23T02:05:25.6976573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015400.xml 2022-11-23T02:05:25.6976805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6977086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6977201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6977487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6977617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6977622Z 2022-11-23T02:05:25.6977688Z Running tests... 2022-11-23T02:05:25.6977885Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6978252Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:05:25.6978257Z 2022-11-23T02:05:25.6978450Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6978519Z Ran 1 test in 0.002s 2022-11-23T02:05:25.6978524Z 2022-11-23T02:05:25.6978590Z OK (skipped=1) 2022-11-23T02:05:25.6978595Z 2022-11-23T02:05:25.6978671Z Generating XML reports... 2022-11-23T02:05:25.6979011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015410.xml 2022-11-23T02:05:25.6979244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6979522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6979640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6979924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6980051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6980056Z 2022-11-23T02:05:25.6980120Z Running tests... 2022-11-23T02:05:25.6980346Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6980571Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25251 2022-11-23T02:05:25.6980723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25252 2022-11-23T02:05:25.6980913Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6981190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6981307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6981587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6981716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6981891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6982167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6982283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6982565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6982691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6982861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6983165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6983466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6983627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6983782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6983963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy58imd6f 2022-11-23T02:05:25.6984153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy58imd6f/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6984328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfns_6yix 2022-11-23T02:05:25.6984515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfns_6yix/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6984609Z ok (7.253s) 2022-11-23T02:05:25.6984613Z 2022-11-23T02:05:25.6984813Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6984882Z Ran 1 test in 7.254s 2022-11-23T02:05:25.6984887Z 2022-11-23T02:05:25.6984940Z OK 2022-11-23T02:05:25.6984945Z 2022-11-23T02:05:25.6985023Z Generating XML reports... 2022-11-23T02:05:25.6985360Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015413.xml 2022-11-23T02:05:25.6985594Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6985873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6985991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6986275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6986407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6986412Z 2022-11-23T02:05:25.6986476Z Running tests... 2022-11-23T02:05:25.6986672Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6986943Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25437 2022-11-23T02:05:25.6987094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25438 2022-11-23T02:05:25.6987285Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6987564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6987682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6987969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6988097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6988268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6988544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6988661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6988946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6989073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6989244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6989547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6989847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6990010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6990163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6990344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhbeftnm 2022-11-23T02:05:25.6990537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhbeftnm/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6990716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lin0pvm 2022-11-23T02:05:25.6990903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lin0pvm/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6991097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6991259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6991564Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:05:25.6991678Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:05:25.6991974Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:05:25.6992080Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:05:25.6992141Z ok (6.346s) 2022-11-23T02:05:25.6992146Z 2022-11-23T02:05:25.6992341Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6992409Z Ran 1 test in 6.347s 2022-11-23T02:05:25.6992414Z 2022-11-23T02:05:25.6992466Z OK 2022-11-23T02:05:25.6992470Z 2022-11-23T02:05:25.6992547Z Generating XML reports... 2022-11-23T02:05:25.6992891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015423.xml 2022-11-23T02:05:25.6993129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.6993445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6993564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6993850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6993975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6993985Z 2022-11-23T02:05:25.6994044Z Running tests... 2022-11-23T02:05:25.6994245Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6994478Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25630 2022-11-23T02:05:25.6994633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25631 2022-11-23T02:05:25.6994823Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.6995104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6995222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6995507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6995636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6995809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.6996130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.6996249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.6996528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.6996658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.6996828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.6997128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6997430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.6997591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.6997810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6hhguok 2022-11-23T02:05:25.6998001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6hhguok/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6998158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.6998331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7k54ag0 2022-11-23T02:05:25.6998520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7k54ag0/_remote_module_non_scriptable.py 2022-11-23T02:05:25.6998684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6998846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.6998906Z ok (4.055s) 2022-11-23T02:05:25.6998911Z 2022-11-23T02:05:25.6999173Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.6999411Z Ran 1 test in 4.056s 2022-11-23T02:05:25.6999417Z 2022-11-23T02:05:25.6999474Z OK 2022-11-23T02:05:25.6999479Z 2022-11-23T02:05:25.6999560Z Generating XML reports... 2022-11-23T02:05:25.6999977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015433.xml 2022-11-23T02:05:25.7000275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7000573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7000699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7001002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7001140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7001145Z 2022-11-23T02:05:25.7001214Z Running tests... 2022-11-23T02:05:25.7001425Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7001670Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25871 2022-11-23T02:05:25.7001832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25872 2022-11-23T02:05:25.7002033Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7002326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7002450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7002746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7002881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7003064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7003383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7003675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7003798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7004099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7004233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7004413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7004733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7004953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7005119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7005312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpratgtz4j 2022-11-23T02:05:25.7005515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpratgtz4j/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7005703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85oo4lo5 2022-11-23T02:05:25.7005898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85oo4lo5/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7006069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.7006243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.7006309Z ok (6.448s) 2022-11-23T02:05:25.7006314Z 2022-11-23T02:05:25.7006528Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7006601Z Ran 1 test in 6.448s 2022-11-23T02:05:25.7006606Z 2022-11-23T02:05:25.7006664Z OK 2022-11-23T02:05:25.7006669Z 2022-11-23T02:05:25.7006744Z Generating XML reports... 2022-11-23T02:05:25.7007138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015440.xml 2022-11-23T02:05:25.7007386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7007682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7007807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7008108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7008247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7008253Z 2022-11-23T02:05:25.7008321Z Running tests... 2022-11-23T02:05:25.7008527Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7008905Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:05:25.7008911Z 2022-11-23T02:05:25.7009115Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7009187Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7009192Z 2022-11-23T02:05:25.7009258Z OK (skipped=1) 2022-11-23T02:05:25.7009263Z 2022-11-23T02:05:25.7009344Z Generating XML reports... 2022-11-23T02:05:25.7009705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015449.xml 2022-11-23T02:05:25.7009949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7010340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7010455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7010741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7010871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7010876Z 2022-11-23T02:05:25.7010942Z Running tests... 2022-11-23T02:05:25.7011138Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7011482Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:05:25.7011487Z 2022-11-23T02:05:25.7011676Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7011773Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7011782Z 2022-11-23T02:05:25.7011841Z OK (skipped=1) 2022-11-23T02:05:25.7011845Z 2022-11-23T02:05:25.7011920Z Generating XML reports... 2022-11-23T02:05:25.7012259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015452.xml 2022-11-23T02:05:25.7012491Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7012766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7012883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7013165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7013294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7013299Z 2022-11-23T02:05:25.7013366Z Running tests... 2022-11-23T02:05:25.7013559Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7013932Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.001s) 2022-11-23T02:05:25.7013937Z 2022-11-23T02:05:25.7014161Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7014232Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7014237Z 2022-11-23T02:05:25.7014300Z OK (skipped=1) 2022-11-23T02:05:25.7014304Z 2022-11-23T02:05:25.7014381Z Generating XML reports... 2022-11-23T02:05:25.7014720Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015455.xml 2022-11-23T02:05:25.7014954Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7015236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7015356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7015640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7015771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7015776Z 2022-11-23T02:05:25.7015842Z Running tests... 2022-11-23T02:05:25.7016033Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7016390Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:05:25.7016401Z 2022-11-23T02:05:25.7016592Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7016659Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7016664Z 2022-11-23T02:05:25.7016730Z OK (skipped=1) 2022-11-23T02:05:25.7016735Z 2022-11-23T02:05:25.7016811Z Generating XML reports... 2022-11-23T02:05:25.7017151Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015458.xml 2022-11-23T02:05:25.7017386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7017663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7017779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7018063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7018190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7018194Z 2022-11-23T02:05:25.7018260Z Running tests... 2022-11-23T02:05:25.7018455Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7018807Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:05:25.7018812Z 2022-11-23T02:05:25.7019005Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7019076Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7019083Z 2022-11-23T02:05:25.7019147Z OK (skipped=1) 2022-11-23T02:05:25.7019151Z 2022-11-23T02:05:25.7019228Z Generating XML reports... 2022-11-23T02:05:25.7019566Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015501.xml 2022-11-23T02:05:25.7019799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7020078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7020197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7020480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7020610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7020619Z 2022-11-23T02:05:25.7020678Z Running tests... 2022-11-23T02:05:25.7020905Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7021135Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26354 2022-11-23T02:05:25.7021285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26355 2022-11-23T02:05:25.7021475Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7021753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7021871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7022157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7022286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7022461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7022733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7022851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7023131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7023259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7023431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7023734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7024032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7024193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7024353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7024534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcscgqwpr 2022-11-23T02:05:25.7024728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcscgqwpr/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7024900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtbgg24g 2022-11-23T02:05:25.7025086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtbgg24g/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7025730Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:05:25.7026311Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:05:25.7026480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.7026676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.7026738Z ok (6.445s) 2022-11-23T02:05:25.7026743Z 2022-11-23T02:05:25.7026946Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7027014Z Ran 1 test in 6.446s 2022-11-23T02:05:25.7027020Z 2022-11-23T02:05:25.7027068Z OK 2022-11-23T02:05:25.7027077Z 2022-11-23T02:05:25.7027149Z Generating XML reports... 2022-11-23T02:05:25.7027495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015504.xml 2022-11-23T02:05:25.7027727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7028009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7028128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7028414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7028544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7028549Z 2022-11-23T02:05:25.7028614Z Running tests... 2022-11-23T02:05:25.7028809Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7029527Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.506s) 2022-11-23T02:05:25.7029535Z 2022-11-23T02:05:25.7029732Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7029800Z Ran 1 test in 0.506s 2022-11-23T02:05:25.7029805Z 2022-11-23T02:05:25.7029868Z OK (skipped=1) 2022-11-23T02:05:25.7029873Z 2022-11-23T02:05:25.7029951Z Generating XML reports... 2022-11-23T02:05:25.7030292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015513.xml 2022-11-23T02:05:25.7030525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7030801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7030919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7031203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7031365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7031370Z 2022-11-23T02:05:25.7031434Z Running tests... 2022-11-23T02:05:25.7031629Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7031975Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:05:25.7031980Z 2022-11-23T02:05:25.7032172Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7032240Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7032245Z 2022-11-23T02:05:25.7032302Z OK (skipped=1) 2022-11-23T02:05:25.7032312Z 2022-11-23T02:05:25.7032386Z Generating XML reports... 2022-11-23T02:05:25.7032726Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015517.xml 2022-11-23T02:05:25.7032960Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7033236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7033355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7033670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7033802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7033807Z 2022-11-23T02:05:25.7033871Z Running tests... 2022-11-23T02:05:25.7034068Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7034404Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:05:25.7034410Z 2022-11-23T02:05:25.7034608Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7034675Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7034680Z 2022-11-23T02:05:25.7034743Z OK (skipped=1) 2022-11-23T02:05:25.7034748Z 2022-11-23T02:05:25.7034824Z Generating XML reports... 2022-11-23T02:05:25.7035164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015520.xml 2022-11-23T02:05:25.7035397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7035671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7035787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7036116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7036246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7036254Z 2022-11-23T02:05:25.7036318Z Running tests... 2022-11-23T02:05:25.7036514Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7036746Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26721 2022-11-23T02:05:25.7036901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26722 2022-11-23T02:05:25.7037091Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7037364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7037482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7037765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7037933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7038105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7038388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7038508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7038790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7038921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7039090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7039394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7039695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7039857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7040014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7040229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qxsrm8p 2022-11-23T02:05:25.7040425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qxsrm8p/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7040603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpty7kblhd 2022-11-23T02:05:25.7040790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpty7kblhd/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7040851Z ok (4.133s) 2022-11-23T02:05:25.7040856Z 2022-11-23T02:05:25.7041056Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7041123Z Ran 1 test in 4.133s 2022-11-23T02:05:25.7041128Z 2022-11-23T02:05:25.7041181Z OK 2022-11-23T02:05:25.7041185Z 2022-11-23T02:05:25.7041262Z Generating XML reports... 2022-11-23T02:05:25.7041604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015523.xml 2022-11-23T02:05:25.7041838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7042115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7042235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7042521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7042650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7042656Z 2022-11-23T02:05:25.7042723Z Running tests... 2022-11-23T02:05:25.7042918Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7043628Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.546s) 2022-11-23T02:05:25.7043634Z 2022-11-23T02:05:25.7043829Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7043899Z Ran 1 test in 0.546s 2022-11-23T02:05:25.7043903Z 2022-11-23T02:05:25.7043968Z OK (skipped=1) 2022-11-23T02:05:25.7043973Z 2022-11-23T02:05:25.7044049Z Generating XML reports... 2022-11-23T02:05:25.7044387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015530.xml 2022-11-23T02:05:25.7044657Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7044933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7045051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7045335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7045464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7045469Z 2022-11-23T02:05:25.7045534Z Running tests... 2022-11-23T02:05:25.7045729Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7046076Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:05:25.7046083Z 2022-11-23T02:05:25.7046276Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7046339Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7046344Z 2022-11-23T02:05:25.7046408Z OK (skipped=1) 2022-11-23T02:05:25.7046412Z 2022-11-23T02:05:25.7046490Z Generating XML reports... 2022-11-23T02:05:25.7046858Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015534.xml 2022-11-23T02:05:25.7047094Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7047371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7047489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7047769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7047898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7047906Z 2022-11-23T02:05:25.7047970Z Running tests... 2022-11-23T02:05:25.7048163Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7048870Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.583s) 2022-11-23T02:05:25.7048875Z 2022-11-23T02:05:25.7049067Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7049135Z Ran 1 test in 0.583s 2022-11-23T02:05:25.7049139Z 2022-11-23T02:05:25.7049203Z OK (skipped=1) 2022-11-23T02:05:25.7049207Z 2022-11-23T02:05:25.7049284Z Generating XML reports... 2022-11-23T02:05:25.7049619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015536.xml 2022-11-23T02:05:25.7049855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7050131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7050251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7050536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7050663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7050668Z 2022-11-23T02:05:25.7050732Z Running tests... 2022-11-23T02:05:25.7050926Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7051163Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27078 2022-11-23T02:05:25.7051341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27079 2022-11-23T02:05:25.7051533Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7051814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7051933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7052217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7052344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7052517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7052795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7052914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7053197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7053369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7053541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7053845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7054146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7054305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7054464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7054647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkg2ajl40 2022-11-23T02:05:25.7054837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkg2ajl40/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7055016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinxc_m_6 2022-11-23T02:05:25.7055199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinxc_m_6/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7055455Z STAGE:2022-11-23 01:55:43 27078:27078 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7055703Z STAGE:2022-11-23 01:55:43 27079:27079 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7055869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.7056029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:05:25.7056290Z STAGE:2022-11-23 01:55:43 27079:27079 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7056538Z STAGE:2022-11-23 01:55:43 27078:27078 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7056802Z STAGE:2022-11-23 01:55:43 27079:27079 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7057064Z STAGE:2022-11-23 01:55:43 27078:27078 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7057231Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:05:25.7057394Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:05:25.7057640Z STAGE:2022-11-23 01:55:43 27078:27078 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7057889Z STAGE:2022-11-23 01:55:43 27078:27078 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7058185Z STAGE:2022-11-23 01:55:43 27078:27078 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7058245Z ok (4.751s) 2022-11-23T02:05:25.7058251Z 2022-11-23T02:05:25.7058447Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7058517Z Ran 1 test in 4.752s 2022-11-23T02:05:25.7058523Z 2022-11-23T02:05:25.7058575Z OK 2022-11-23T02:05:25.7058580Z 2022-11-23T02:05:25.7058655Z Generating XML reports... 2022-11-23T02:05:25.7058999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015540.xml 2022-11-23T02:05:25.7059230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7059507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7059624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7059909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7060040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7060045Z 2022-11-23T02:05:25.7060111Z Running tests... 2022-11-23T02:05:25.7060333Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7060568Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27275 2022-11-23T02:05:25.7060718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27276 2022-11-23T02:05:25.7060910Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7061189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7061311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7061594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7061722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7061895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7062198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7062473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7062589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7062870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7063002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7063170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7063473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7063631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7063790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7063969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9z5vjtv 2022-11-23T02:05:25.7064157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9z5vjtv/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7064331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mxa_f4h 2022-11-23T02:05:25.7064548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mxa_f4h/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7064608Z ok (4.443s) 2022-11-23T02:05:25.7064613Z 2022-11-23T02:05:25.7064810Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7064879Z Ran 1 test in 4.443s 2022-11-23T02:05:25.7064884Z 2022-11-23T02:05:25.7064939Z OK 2022-11-23T02:05:25.7064943Z 2022-11-23T02:05:25.7065020Z Generating XML reports... 2022-11-23T02:05:25.7065362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015548.xml 2022-11-23T02:05:25.7065594Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7065870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7065987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7066269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7066401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7066406Z 2022-11-23T02:05:25.7066470Z Running tests... 2022-11-23T02:05:25.7066698Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7066939Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27462 2022-11-23T02:05:25.7067089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27463 2022-11-23T02:05:25.7067278Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7067557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7067675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7067960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7068087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7068260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7068536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7068653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7068933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7069061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7069231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7069533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7069829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7069990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7070149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7070330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3ukhf5q 2022-11-23T02:05:25.7070520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3ukhf5q/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7070696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj9m4ud6 2022-11-23T02:05:25.7070883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj9m4ud6/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7070981Z ok (4.043s) 2022-11-23T02:05:25.7070986Z 2022-11-23T02:05:25.7071184Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7071253Z Ran 1 test in 4.043s 2022-11-23T02:05:25.7071257Z 2022-11-23T02:05:25.7071309Z OK 2022-11-23T02:05:25.7071313Z 2022-11-23T02:05:25.7071394Z Generating XML reports... 2022-11-23T02:05:25.7071736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015555.xml 2022-11-23T02:05:25.7071968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7072246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7072363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7072647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7072776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7072781Z 2022-11-23T02:05:25.7072841Z Running tests... 2022-11-23T02:05:25.7073038Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7073782Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:05:25.7073793Z 2022-11-23T02:05:25.7073986Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7074049Z Ran 1 test in 0.509s 2022-11-23T02:05:25.7074055Z 2022-11-23T02:05:25.7074118Z OK (skipped=1) 2022-11-23T02:05:25.7074125Z 2022-11-23T02:05:25.7074205Z Generating XML reports... 2022-11-23T02:05:25.7074542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015602.xml 2022-11-23T02:05:25.7074776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7075054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7075175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7075459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7075588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7075593Z 2022-11-23T02:05:25.7075657Z Running tests... 2022-11-23T02:05:25.7075852Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7076591Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.524s) 2022-11-23T02:05:25.7076599Z 2022-11-23T02:05:25.7076795Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7076865Z Ran 1 test in 0.524s 2022-11-23T02:05:25.7076870Z 2022-11-23T02:05:25.7076934Z OK (skipped=1) 2022-11-23T02:05:25.7076939Z 2022-11-23T02:05:25.7077016Z Generating XML reports... 2022-11-23T02:05:25.7077358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015606.xml 2022-11-23T02:05:25.7077590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7077869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7078029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7078316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7078450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7078456Z 2022-11-23T02:05:25.7078522Z Running tests... 2022-11-23T02:05:25.7078717Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7079437Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.574s) 2022-11-23T02:05:25.7079445Z 2022-11-23T02:05:25.7079642Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7079709Z Ran 1 test in 0.575s 2022-11-23T02:05:25.7079714Z 2022-11-23T02:05:25.7079778Z OK (skipped=1) 2022-11-23T02:05:25.7079783Z 2022-11-23T02:05:25.7079856Z Generating XML reports... 2022-11-23T02:05:25.7080234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015609.xml 2022-11-23T02:05:25.7080471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7080747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7080866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7081152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7081284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7081289Z 2022-11-23T02:05:25.7081352Z Running tests... 2022-11-23T02:05:25.7081545Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7081917Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:05:25.7081923Z 2022-11-23T02:05:25.7082117Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7082185Z Ran 1 test in 0.003s 2022-11-23T02:05:25.7082190Z 2022-11-23T02:05:25.7082253Z OK (skipped=1) 2022-11-23T02:05:25.7082258Z 2022-11-23T02:05:25.7082335Z Generating XML reports... 2022-11-23T02:05:25.7082671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015613.xml 2022-11-23T02:05:25.7082903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7083179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7083295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7083578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7083706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7083711Z 2022-11-23T02:05:25.7083777Z Running tests... 2022-11-23T02:05:25.7083972Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7084206Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27877 2022-11-23T02:05:25.7084355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27878 2022-11-23T02:05:25.7084577Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7084854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7084974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7085257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7085387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7085557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7085862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7086139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7086258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7086543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7086702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7086874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7087176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7087333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7087493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7087779Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:05:25.7087970Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:05:25.7088150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfa8iakx 2022-11-23T02:05:25.7088341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfa8iakx/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7088626Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:05:25.7088814Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:05:25.7088991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rnxjk5h 2022-11-23T02:05:25.7089176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rnxjk5h/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7089235Z ok (4.542s) 2022-11-23T02:05:25.7089240Z 2022-11-23T02:05:25.7089437Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7089506Z Ran 1 test in 4.542s 2022-11-23T02:05:25.7089512Z 2022-11-23T02:05:25.7089564Z OK 2022-11-23T02:05:25.7089568Z 2022-11-23T02:05:25.7089644Z Generating XML reports... 2022-11-23T02:05:25.7089984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015616.xml 2022-11-23T02:05:25.7090217Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7090494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7090610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7090898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7091027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7091032Z 2022-11-23T02:05:25.7091132Z Running tests... 2022-11-23T02:05:25.7091330Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7091558Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28060 2022-11-23T02:05:25.7091713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28061 2022-11-23T02:05:25.7091902Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7092179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7092297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7092581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7092708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7092881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7093179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7093488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7093606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7093893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7094022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7094192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7094492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7094651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7094809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7094980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7095149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7095446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7095739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7095799Z ok (3.939s) 2022-11-23T02:05:25.7095804Z 2022-11-23T02:05:25.7095999Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7096070Z Ran 1 test in 3.939s 2022-11-23T02:05:25.7096075Z 2022-11-23T02:05:25.7096128Z OK 2022-11-23T02:05:25.7096132Z 2022-11-23T02:05:25.7096209Z Generating XML reports... 2022-11-23T02:05:25.7096556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015624.xml 2022-11-23T02:05:25.7096789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7097068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7097186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7097464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7097592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7097601Z 2022-11-23T02:05:25.7097697Z Running tests... 2022-11-23T02:05:25.7097893Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7098114Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28249 2022-11-23T02:05:25.7098268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28250 2022-11-23T02:05:25.7098461Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7098735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7098851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7099135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7099266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7099439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7099738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7100045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7100163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7100448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7100576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7100745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7101047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7101209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7101366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7101537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7101704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7102002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7102294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7102354Z ok (3.839s) 2022-11-23T02:05:25.7102359Z 2022-11-23T02:05:25.7102556Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7102628Z Ran 1 test in 3.839s 2022-11-23T02:05:25.7102633Z 2022-11-23T02:05:25.7102687Z OK 2022-11-23T02:05:25.7102691Z 2022-11-23T02:05:25.7102769Z Generating XML reports... 2022-11-23T02:05:25.7103114Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015631.xml 2022-11-23T02:05:25.7103349Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7103624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7103741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7104027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7104156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7104161Z 2022-11-23T02:05:25.7104225Z Running tests... 2022-11-23T02:05:25.7104471Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7105180Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.518s) 2022-11-23T02:05:25.7105186Z 2022-11-23T02:05:25.7105381Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7105449Z Ran 1 test in 0.519s 2022-11-23T02:05:25.7105454Z 2022-11-23T02:05:25.7105518Z OK (skipped=1) 2022-11-23T02:05:25.7105522Z 2022-11-23T02:05:25.7105601Z Generating XML reports... 2022-11-23T02:05:25.7105941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015638.xml 2022-11-23T02:05:25.7106179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7106457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7106574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7106891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7107021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7107025Z 2022-11-23T02:05:25.7107085Z Running tests... 2022-11-23T02:05:25.7107283Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7107620Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:05:25.7107626Z 2022-11-23T02:05:25.7107819Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7107890Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7107895Z 2022-11-23T02:05:25.7107959Z OK (skipped=1) 2022-11-23T02:05:25.7107964Z 2022-11-23T02:05:25.7108041Z Generating XML reports... 2022-11-23T02:05:25.7108383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015641.xml 2022-11-23T02:05:25.7108616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7108891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7109008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7109291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7109418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7109425Z 2022-11-23T02:05:25.7109491Z Running tests... 2022-11-23T02:05:25.7109685Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7109919Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28554 2022-11-23T02:05:25.7110072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28555 2022-11-23T02:05:25.7110264Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7110538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7110654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7110938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7111102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7111276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7111548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7111667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7111951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7112082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7112253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7112556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7112855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7113019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7113176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7113268Z ok (3.827s) 2022-11-23T02:05:25.7113274Z 2022-11-23T02:05:25.7113471Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7113538Z Ran 1 test in 3.828s 2022-11-23T02:05:25.7113543Z 2022-11-23T02:05:25.7113595Z OK 2022-11-23T02:05:25.7113599Z 2022-11-23T02:05:25.7113675Z Generating XML reports... 2022-11-23T02:05:25.7114019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015644.xml 2022-11-23T02:05:25.7114251Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7114527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7114643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7114926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7115056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7115061Z 2022-11-23T02:05:25.7115126Z Running tests... 2022-11-23T02:05:25.7115320Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7115531Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28737 2022-11-23T02:05:25.7115681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28738 2022-11-23T02:05:25.7115870Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7116186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7116302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7116588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7116715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7116886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7117193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7117468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7117585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7117911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7118039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7118211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7118510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7118669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7118924Z STAGE:2022-11-23 01:56:53 28738:28738 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7119080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7119332Z STAGE:2022-11-23 01:56:53 28737:28737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7119588Z STAGE:2022-11-23 01:56:53 28738:28738 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7119850Z STAGE:2022-11-23 01:56:53 28738:28738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7120137Z STAGE:2022-11-23 01:56:53 28737:28737 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7120401Z STAGE:2022-11-23 01:56:53 28737:28737 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7120645Z STAGE:2022-11-23 01:56:53 28738:28738 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7120880Z STAGE:2022-11-23 01:56:53 28737:28737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7121129Z STAGE:2022-11-23 01:56:53 28737:28737 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7121385Z STAGE:2022-11-23 01:56:53 28737:28737 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7121635Z STAGE:2022-11-23 01:56:53 28738:28738 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7121893Z STAGE:2022-11-23 01:56:53 28738:28738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7121952Z ok (3.838s) 2022-11-23T02:05:25.7121959Z 2022-11-23T02:05:25.7122158Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7122226Z Ran 1 test in 3.839s 2022-11-23T02:05:25.7122231Z 2022-11-23T02:05:25.7122285Z OK 2022-11-23T02:05:25.7122290Z 2022-11-23T02:05:25.7122366Z Generating XML reports... 2022-11-23T02:05:25.7122708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015651.xml 2022-11-23T02:05:25.7122941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7123220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7123341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7123625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7123756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7123760Z 2022-11-23T02:05:25.7123825Z Running tests... 2022-11-23T02:05:25.7124020Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7124244Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28926 2022-11-23T02:05:25.7124396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28927 2022-11-23T02:05:25.7124588Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7124899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7125012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7125298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7125427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7125598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7125873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7125989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7126275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7126405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7126577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7126914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7127217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7127374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7127531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7127591Z ok (3.762s) 2022-11-23T02:05:25.7127596Z 2022-11-23T02:05:25.7127794Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7127863Z Ran 1 test in 3.763s 2022-11-23T02:05:25.7127870Z 2022-11-23T02:05:25.7127924Z OK 2022-11-23T02:05:25.7127928Z 2022-11-23T02:05:25.7128004Z Generating XML reports... 2022-11-23T02:05:25.7128346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015658.xml 2022-11-23T02:05:25.7128584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7128861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7132724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7133057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7133194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7133200Z 2022-11-23T02:05:25.7133267Z Running tests... 2022-11-23T02:05:25.7133467Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7133657Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-11-23T02:05:25.7133666Z 2022-11-23T02:05:25.7133856Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7133923Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7133929Z 2022-11-23T02:05:25.7133994Z OK (skipped=1) 2022-11-23T02:05:25.7133999Z 2022-11-23T02:05:25.7134075Z Generating XML reports... 2022-11-23T02:05:25.7134422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015705.xml 2022-11-23T02:05:25.7134654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7134932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7135051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7135419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7135549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7135555Z 2022-11-23T02:05:25.7135619Z Running tests... 2022-11-23T02:05:25.7135820Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7136049Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29167 2022-11-23T02:05:25.7136199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29168 2022-11-23T02:05:25.7136390Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7136667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7136788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7137071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7137200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7137411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7137691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7137805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7138088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7138214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7138381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7138685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7138981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7139142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7139300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7139468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7139634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7139929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7140222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7140475Z STAGE:2022-11-23 01:57:10 29167:29167 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7140721Z STAGE:2022-11-23 01:57:10 29168:29168 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7140974Z STAGE:2022-11-23 01:57:10 29168:29168 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7141236Z STAGE:2022-11-23 01:57:10 29168:29168 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7141485Z STAGE:2022-11-23 01:57:10 29167:29167 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7141746Z STAGE:2022-11-23 01:57:10 29167:29167 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7141986Z STAGE:2022-11-23 01:57:10 29168:29168 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7142262Z STAGE:2022-11-23 01:57:10 29167:29167 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7142510Z STAGE:2022-11-23 01:57:10 29167:29167 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7142770Z STAGE:2022-11-23 01:57:10 29167:29167 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7143020Z STAGE:2022-11-23 01:57:10 29168:29168 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7143277Z STAGE:2022-11-23 01:57:10 29168:29168 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7143333Z ok (4.037s) 2022-11-23T02:05:25.7143341Z 2022-11-23T02:05:25.7143534Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7143603Z Ran 1 test in 4.038s 2022-11-23T02:05:25.7143608Z 2022-11-23T02:05:25.7143660Z OK 2022-11-23T02:05:25.7143665Z 2022-11-23T02:05:25.7143744Z Generating XML reports... 2022-11-23T02:05:25.7144086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015707.xml 2022-11-23T02:05:25.7144320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7144639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7144758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7145046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7145176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7145181Z 2022-11-23T02:05:25.7145247Z Running tests... 2022-11-23T02:05:25.7145443Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7145668Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29362 2022-11-23T02:05:25.7145825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29363 2022-11-23T02:05:25.7146016Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7146295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7146412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7146697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7146821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7146994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7147296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7147577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7147695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7147982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7148110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7148283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7148586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7148745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7148940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7149045Z skip: Skipped due to small world size. (3.941s) 2022-11-23T02:05:25.7149050Z 2022-11-23T02:05:25.7149251Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7149318Z Ran 1 test in 3.941s 2022-11-23T02:05:25.7149326Z 2022-11-23T02:05:25.7149390Z OK (skipped=1) 2022-11-23T02:05:25.7149395Z 2022-11-23T02:05:25.7149471Z Generating XML reports... 2022-11-23T02:05:25.7149815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015714.xml 2022-11-23T02:05:25.7150050Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7150327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7150445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7150729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7150858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7150863Z 2022-11-23T02:05:25.7150930Z Running tests... 2022-11-23T02:05:25.7151153Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7151383Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29545 2022-11-23T02:05:25.7151534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29546 2022-11-23T02:05:25.7151724Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7152001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7152121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7152404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7152534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7152708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7152987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7153104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7153386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7153513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7153684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7153988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7154290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7154455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7154613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7154674Z ok (3.852s) 2022-11-23T02:05:25.7154679Z 2022-11-23T02:05:25.7154877Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7154946Z Ran 1 test in 3.852s 2022-11-23T02:05:25.7154951Z 2022-11-23T02:05:25.7155003Z OK 2022-11-23T02:05:25.7155008Z 2022-11-23T02:05:25.7155080Z Generating XML reports... 2022-11-23T02:05:25.7155421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015721.xml 2022-11-23T02:05:25.7155691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7155968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7156169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7156454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7156582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7156587Z 2022-11-23T02:05:25.7156650Z Running tests... 2022-11-23T02:05:25.7156847Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7157544Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.522s) 2022-11-23T02:05:25.7157552Z 2022-11-23T02:05:25.7157823Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7157894Z Ran 1 test in 0.522s 2022-11-23T02:05:25.7157899Z 2022-11-23T02:05:25.7157963Z OK (skipped=1) 2022-11-23T02:05:25.7157967Z 2022-11-23T02:05:25.7158044Z Generating XML reports... 2022-11-23T02:05:25.7158389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015728.xml 2022-11-23T02:05:25.7158621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7158895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7159014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7159297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7159425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7159430Z 2022-11-23T02:05:25.7159498Z Running tests... 2022-11-23T02:05:25.7159694Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7159916Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29786 2022-11-23T02:05:25.7160065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29787 2022-11-23T02:05:25.7160256Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7160532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7160647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7160931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7161061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7161233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7161509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7161625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7161904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7162030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7162242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7162547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7162849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7163010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7163166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7163338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7163636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7163805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7164102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7164163Z ok (3.928s) 2022-11-23T02:05:25.7164168Z 2022-11-23T02:05:25.7164395Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7164464Z Ran 1 test in 3.929s 2022-11-23T02:05:25.7164469Z 2022-11-23T02:05:25.7164524Z OK 2022-11-23T02:05:25.7164529Z 2022-11-23T02:05:25.7164604Z Generating XML reports... 2022-11-23T02:05:25.7164945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015732.xml 2022-11-23T02:05:25.7165181Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7165459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7165578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7165862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7165991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7165996Z 2022-11-23T02:05:25.7166063Z Running tests... 2022-11-23T02:05:25.7166260Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7166483Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29975 2022-11-23T02:05:25.7166633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29976 2022-11-23T02:05:25.7166822Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7167095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7167214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7167496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7167625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7167794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7168068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7168183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7168470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7168599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7168805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7169105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7169403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7169559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7169716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7169776Z ok (3.835s) 2022-11-23T02:05:25.7169781Z 2022-11-23T02:05:25.7169978Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7170045Z Ran 1 test in 3.835s 2022-11-23T02:05:25.7170050Z 2022-11-23T02:05:25.7170102Z OK 2022-11-23T02:05:25.7170107Z 2022-11-23T02:05:25.7170183Z Generating XML reports... 2022-11-23T02:05:25.7170526Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015739.xml 2022-11-23T02:05:25.7170758Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7171152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7171271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7171556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7171685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7171690Z 2022-11-23T02:05:25.7171756Z Running tests... 2022-11-23T02:05:25.7171952Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7172170Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30158 2022-11-23T02:05:25.7172324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30159 2022-11-23T02:05:25.7172513Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7172788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7172903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7173184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7173312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7173477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7173778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7174056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7174174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7174461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7174588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7174758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7175061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7175220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7175376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7175470Z ok (4.051s) 2022-11-23T02:05:25.7175475Z 2022-11-23T02:05:25.7175672Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7175741Z Ran 1 test in 4.051s 2022-11-23T02:05:25.7175746Z 2022-11-23T02:05:25.7175800Z OK 2022-11-23T02:05:25.7175806Z 2022-11-23T02:05:25.7175883Z Generating XML reports... 2022-11-23T02:05:25.7176223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015746.xml 2022-11-23T02:05:25.7176454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7176730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7176848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7177131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7177261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7177266Z 2022-11-23T02:05:25.7177331Z Running tests... 2022-11-23T02:05:25.7177523Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7177786Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30341 2022-11-23T02:05:25.7177940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30342 2022-11-23T02:05:25.7178128Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7178400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7178516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7178803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7178931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7179104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7179404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7179678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7179796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7180082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7180213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7180385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7180684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7180848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7181004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7181171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7181338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7181637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7181930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7182022Z ok (3.754s) 2022-11-23T02:05:25.7182027Z 2022-11-23T02:05:25.7182219Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7182287Z Ran 1 test in 3.754s 2022-11-23T02:05:25.7182292Z 2022-11-23T02:05:25.7182344Z OK 2022-11-23T02:05:25.7182351Z 2022-11-23T02:05:25.7182427Z Generating XML reports... 2022-11-23T02:05:25.7182765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015753.xml 2022-11-23T02:05:25.7182996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7183271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7183387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7183671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7183800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7183805Z 2022-11-23T02:05:25.7183869Z Running tests... 2022-11-23T02:05:25.7184064Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7184327Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30530 2022-11-23T02:05:25.7184480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30531 2022-11-23T02:05:25.7184671Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7184948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7185065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7185351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7185481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7185654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7185954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7186232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7186347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7186629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7186757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7186930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7187230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7187389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7187547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7187714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7187880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7188177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7188472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7188565Z ok (4.040s) 2022-11-23T02:05:25.7188571Z 2022-11-23T02:05:25.7188768Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7188837Z Ran 1 test in 4.040s 2022-11-23T02:05:25.7188842Z 2022-11-23T02:05:25.7188894Z OK 2022-11-23T02:05:25.7188899Z 2022-11-23T02:05:25.7188978Z Generating XML reports... 2022-11-23T02:05:25.7189318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015800.xml 2022-11-23T02:05:25.7189550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7189827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7189946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7190232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7190364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7190369Z 2022-11-23T02:05:25.7190433Z Running tests... 2022-11-23T02:05:25.7190625Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7190981Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:05:25.7190991Z 2022-11-23T02:05:25.7191179Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7191247Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7191252Z 2022-11-23T02:05:25.7191315Z OK (skipped=1) 2022-11-23T02:05:25.7191319Z 2022-11-23T02:05:25.7191399Z Generating XML reports... 2022-11-23T02:05:25.7191738Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015807.xml 2022-11-23T02:05:25.7191973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7192249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7192367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7192654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7192782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7192786Z 2022-11-23T02:05:25.7192851Z Running tests... 2022-11-23T02:05:25.7193045Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7193262Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30777 2022-11-23T02:05:25.7193413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30778 2022-11-23T02:05:25.7193604Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7193880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7193997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7194280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7194407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7194578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7194881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7195156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7195300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7195586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7195716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7195886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7196232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7196391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7196550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7196611Z ok (4.028s) 2022-11-23T02:05:25.7196616Z 2022-11-23T02:05:25.7196816Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7196885Z Ran 1 test in 4.029s 2022-11-23T02:05:25.7196890Z 2022-11-23T02:05:25.7196945Z OK 2022-11-23T02:05:25.7196950Z 2022-11-23T02:05:25.7197027Z Generating XML reports... 2022-11-23T02:05:25.7197409Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015810.xml 2022-11-23T02:05:25.7197645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7197922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7198040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7198323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7198449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7198457Z 2022-11-23T02:05:25.7198522Z Running tests... 2022-11-23T02:05:25.7198719Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7198933Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30960 2022-11-23T02:05:25.7199080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30961 2022-11-23T02:05:25.7199271Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7199549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7199667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7199953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7200084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7200255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7200531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7200650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7200934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7201060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7201231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7201532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7201832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7202027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7202184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7202247Z ok (3.960s) 2022-11-23T02:05:25.7202253Z 2022-11-23T02:05:25.7202451Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7202519Z Ran 1 test in 3.961s 2022-11-23T02:05:25.7202524Z 2022-11-23T02:05:25.7202577Z OK 2022-11-23T02:05:25.7202581Z 2022-11-23T02:05:25.7202656Z Generating XML reports... 2022-11-23T02:05:25.7202997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015817.xml 2022-11-23T02:05:25.7203224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7203500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7203619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7203901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7204068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7204073Z 2022-11-23T02:05:25.7204141Z Running tests... 2022-11-23T02:05:25.7204337Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7204572Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31143 2022-11-23T02:05:25.7204725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31144 2022-11-23T02:05:25.7204914Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7205190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7205308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7205592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7205720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7205891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7206168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7206286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7206567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7206696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7206867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7207172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7207472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7207632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7207886Z STAGE:2022-11-23 01:58:26 31144:31144 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7208037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7208291Z STAGE:2022-11-23 01:58:26 31143:31143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7208591Z STAGE:2022-11-23 01:58:26 31144:31144 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7208856Z STAGE:2022-11-23 01:58:26 31144:31144 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7209110Z STAGE:2022-11-23 01:58:26 31143:31143 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7209371Z STAGE:2022-11-23 01:58:26 31143:31143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7209434Z ok (3.940s) 2022-11-23T02:05:25.7209440Z 2022-11-23T02:05:25.7209637Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7209704Z Ran 1 test in 3.941s 2022-11-23T02:05:25.7209710Z 2022-11-23T02:05:25.7209762Z OK 2022-11-23T02:05:25.7209766Z 2022-11-23T02:05:25.7209842Z Generating XML reports... 2022-11-23T02:05:25.7210186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015824.xml 2022-11-23T02:05:25.7210422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7210703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7210855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7211144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7211275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7211280Z 2022-11-23T02:05:25.7211345Z Running tests... 2022-11-23T02:05:25.7211544Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7211776Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31332 2022-11-23T02:05:25.7211930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31333 2022-11-23T02:05:25.7212120Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7212392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7212510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7212794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7212920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7213089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7213365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7213483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7213768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7213896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7214069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7214370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7214666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7214824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7215079Z STAGE:2022-11-23 01:58:33 31333:31333 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7215267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7215515Z STAGE:2022-11-23 01:58:33 31332:31332 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7215767Z STAGE:2022-11-23 01:58:33 31333:31333 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7216034Z STAGE:2022-11-23 01:58:33 31333:31333 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7216288Z STAGE:2022-11-23 01:58:33 31332:31332 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7216547Z STAGE:2022-11-23 01:58:33 31332:31332 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7216607Z ok (3.941s) 2022-11-23T02:05:25.7216612Z 2022-11-23T02:05:25.7216810Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7216873Z Ran 1 test in 3.941s 2022-11-23T02:05:25.7216885Z 2022-11-23T02:05:25.7216933Z OK 2022-11-23T02:05:25.7216937Z 2022-11-23T02:05:25.7217012Z Generating XML reports... 2022-11-23T02:05:25.7217353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015831.xml 2022-11-23T02:05:25.7217620Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7217903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7218020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7218304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7218433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7218438Z 2022-11-23T02:05:25.7218503Z Running tests... 2022-11-23T02:05:25.7218697Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7219046Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:05:25.7219051Z 2022-11-23T02:05:25.7219245Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7219314Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7219320Z 2022-11-23T02:05:25.7219384Z OK (skipped=1) 2022-11-23T02:05:25.7219389Z 2022-11-23T02:05:25.7219470Z Generating XML reports... 2022-11-23T02:05:25.7219812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015838.xml 2022-11-23T02:05:25.7220051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7220329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7220446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7220731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7220860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7220865Z 2022-11-23T02:05:25.7220930Z Running tests... 2022-11-23T02:05:25.7221122Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7221492Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:05:25.7221502Z 2022-11-23T02:05:25.7221690Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7221758Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7221763Z 2022-11-23T02:05:25.7221827Z OK (skipped=1) 2022-11-23T02:05:25.7221831Z 2022-11-23T02:05:25.7221908Z Generating XML reports... 2022-11-23T02:05:25.7222286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015841.xml 2022-11-23T02:05:25.7222518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7222794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7222911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7223196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7223325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7223330Z 2022-11-23T02:05:25.7223395Z Running tests... 2022-11-23T02:05:25.7223588Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7223830Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31637 2022-11-23T02:05:25.7223984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31638 2022-11-23T02:05:25.7224173Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7224480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7224599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7224884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7225013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7225185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7225488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7225766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7225879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7226163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7226290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7226459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7226761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7226922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7227079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7227184Z skip: Skipped due to small world size. (3.731s) 2022-11-23T02:05:25.7227190Z 2022-11-23T02:05:25.7227385Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7227454Z Ran 1 test in 3.732s 2022-11-23T02:05:25.7227459Z 2022-11-23T02:05:25.7227524Z OK (skipped=1) 2022-11-23T02:05:25.7227529Z 2022-11-23T02:05:25.7227604Z Generating XML reports... 2022-11-23T02:05:25.7227949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015844.xml 2022-11-23T02:05:25.7228181Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7228462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7228580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7228898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7229033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7229038Z 2022-11-23T02:05:25.7229104Z Running tests... 2022-11-23T02:05:25.7229300Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7229535Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31820 2022-11-23T02:05:25.7229688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31821 2022-11-23T02:05:25.7229882Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7230152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7230273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7230559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7230686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7230891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7231171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7231287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7231569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7231696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7231869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7232173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7232471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7232632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7232790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7232949Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:05:25.7233010Z ok (6.059s) 2022-11-23T02:05:25.7233015Z 2022-11-23T02:05:25.7233212Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7233280Z Ran 1 test in 6.059s 2022-11-23T02:05:25.7233285Z 2022-11-23T02:05:25.7233338Z OK 2022-11-23T02:05:25.7233345Z 2022-11-23T02:05:25.7233421Z Generating XML reports... 2022-11-23T02:05:25.7233762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015850.xml 2022-11-23T02:05:25.7233992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7234266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7234383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7234667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7234795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7234799Z 2022-11-23T02:05:25.7234865Z Running tests... 2022-11-23T02:05:25.7235060Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7235337Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32003 2022-11-23T02:05:25.7235488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32004 2022-11-23T02:05:25.7235679Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7235957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7236120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7236402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7236532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7236702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7236977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7237093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7237412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7237544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7237716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7238016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7238314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7238473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7238634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7238798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7238967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7239262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7239554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7239702Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-11-23T02:05:25.7239820Z No ranks successfully processed in monitoredBarrier. 2022-11-23T02:05:25.7239976Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-11-23T02:05:25.7240038Z ok (3.950s) 2022-11-23T02:05:25.7240043Z 2022-11-23T02:05:25.7240241Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7240308Z Ran 1 test in 3.951s 2022-11-23T02:05:25.7240314Z 2022-11-23T02:05:25.7240366Z OK 2022-11-23T02:05:25.7240371Z 2022-11-23T02:05:25.7240446Z Generating XML reports... 2022-11-23T02:05:25.7240791Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015859.xml 2022-11-23T02:05:25.7241026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7241304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7241422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7241708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7241872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7241877Z 2022-11-23T02:05:25.7241941Z Running tests... 2022-11-23T02:05:25.7242139Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7242383Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32192 2022-11-23T02:05:25.7242533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32193 2022-11-23T02:05:25.7242721Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7242999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7243115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7243398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7243530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7243701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7244014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7244132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7244416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7244544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7244712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7245015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7245318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7245478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7245636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7245805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7245970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7246270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7246571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7246730Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-11-23T02:05:25.7246791Z ok (3.941s) 2022-11-23T02:05:25.7246797Z 2022-11-23T02:05:25.7246996Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7247065Z Ran 1 test in 3.941s 2022-11-23T02:05:25.7247070Z 2022-11-23T02:05:25.7247120Z OK 2022-11-23T02:05:25.7247130Z 2022-11-23T02:05:25.7247201Z Generating XML reports... 2022-11-23T02:05:25.7247545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015906.xml 2022-11-23T02:05:25.7247781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7248054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7248170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7248487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7248614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7248620Z 2022-11-23T02:05:25.7248684Z Running tests... 2022-11-23T02:05:25.7248883Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7249125Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32381 2022-11-23T02:05:25.7249278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32382 2022-11-23T02:05:25.7249469Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7249745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7249866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7250148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7250274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7250476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7250778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7251053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7251171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7251457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7251584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7251752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7252053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7252213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7252371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7252477Z skip: Skipped due to small world size. (3.856s) 2022-11-23T02:05:25.7252482Z 2022-11-23T02:05:25.7252681Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7252749Z Ran 1 test in 3.856s 2022-11-23T02:05:25.7252753Z 2022-11-23T02:05:25.7252817Z OK (skipped=1) 2022-11-23T02:05:25.7252821Z 2022-11-23T02:05:25.7252900Z Generating XML reports... 2022-11-23T02:05:25.7253243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015913.xml 2022-11-23T02:05:25.7253475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7253754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7253873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7254156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7254286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7254291Z 2022-11-23T02:05:25.7254356Z Running tests... 2022-11-23T02:05:25.7254553Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7254861Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:05:25.7254902Z 2022-11-23T02:05:25.7255098Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7255167Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7255172Z 2022-11-23T02:05:25.7255234Z OK (skipped=1) 2022-11-23T02:05:25.7255239Z 2022-11-23T02:05:25.7255316Z Generating XML reports... 2022-11-23T02:05:25.7255654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015920.xml 2022-11-23T02:05:25.7255887Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7256163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7256283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7256566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7256700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7256705Z 2022-11-23T02:05:25.7256770Z Running tests... 2022-11-23T02:05:25.7256966Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7257326Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:05:25.7257333Z 2022-11-23T02:05:25.7257528Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7257597Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7257602Z 2022-11-23T02:05:25.7257667Z OK (skipped=1) 2022-11-23T02:05:25.7257671Z 2022-11-23T02:05:25.7257747Z Generating XML reports... 2022-11-23T02:05:25.7258087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015923.xml 2022-11-23T02:05:25.7258319Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7258598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7258714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7259000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7259128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7259133Z 2022-11-23T02:05:25.7259197Z Running tests... 2022-11-23T02:05:25.7259392Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7259696Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:05:25.7259701Z 2022-11-23T02:05:25.7259892Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7259957Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7259966Z 2022-11-23T02:05:25.7260024Z OK (skipped=1) 2022-11-23T02:05:25.7260028Z 2022-11-23T02:05:25.7260106Z Generating XML reports... 2022-11-23T02:05:25.7260443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015926.xml 2022-11-23T02:05:25.7260679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7260955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7261074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7261355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7261482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7261487Z 2022-11-23T02:05:25.7261585Z Running tests... 2022-11-23T02:05:25.7261780Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7262082Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:05:25.7262088Z 2022-11-23T02:05:25.7262283Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7262351Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7262356Z 2022-11-23T02:05:25.7262418Z OK (skipped=1) 2022-11-23T02:05:25.7262423Z 2022-11-23T02:05:25.7262499Z Generating XML reports... 2022-11-23T02:05:25.7262836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015929.xml 2022-11-23T02:05:25.7263066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7263342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7263463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7263745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7263872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7263911Z 2022-11-23T02:05:25.7263976Z Running tests... 2022-11-23T02:05:25.7264166Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7264379Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s) 2022-11-23T02:05:25.7264389Z 2022-11-23T02:05:25.7264576Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7264644Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7264649Z 2022-11-23T02:05:25.7264713Z OK (skipped=1) 2022-11-23T02:05:25.7264717Z 2022-11-23T02:05:25.7264793Z Generating XML reports... 2022-11-23T02:05:25.7265135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015932.xml 2022-11-23T02:05:25.7265365Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7265642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7265759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7266045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7266175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7266180Z 2022-11-23T02:05:25.7266244Z Running tests... 2022-11-23T02:05:25.7266440Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7266613Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:05:25.7266621Z 2022-11-23T02:05:25.7266812Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7266879Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7266884Z 2022-11-23T02:05:25.7266947Z OK (skipped=1) 2022-11-23T02:05:25.7266951Z 2022-11-23T02:05:25.7267029Z Generating XML reports... 2022-11-23T02:05:25.7267364Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015935.xml 2022-11-23T02:05:25.7267596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7267872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7267989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7268267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7268430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7268435Z 2022-11-23T02:05:25.7268502Z Running tests... 2022-11-23T02:05:25.7268698Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7268890Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:05:25.7268896Z 2022-11-23T02:05:25.7269086Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7269153Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7269158Z 2022-11-23T02:05:25.7269220Z OK (skipped=1) 2022-11-23T02:05:25.7269224Z 2022-11-23T02:05:25.7269300Z Generating XML reports... 2022-11-23T02:05:25.7269637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015938.xml 2022-11-23T02:05:25.7269867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7270144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7270261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7270579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7270707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7270712Z 2022-11-23T02:05:25.7270779Z Running tests... 2022-11-23T02:05:25.7270974Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7271196Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:05:25.7271201Z 2022-11-23T02:05:25.7271392Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7271467Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7271472Z 2022-11-23T02:05:25.7271535Z OK (skipped=1) 2022-11-23T02:05:25.7271539Z 2022-11-23T02:05:25.7271615Z Generating XML reports... 2022-11-23T02:05:25.7271951Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015941.xml 2022-11-23T02:05:25.7272183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7272460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7272576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7272859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7272986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7272990Z 2022-11-23T02:05:25.7273056Z Running tests... 2022-11-23T02:05:25.7273249Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7273505Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33028 2022-11-23T02:05:25.7273661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33029 2022-11-23T02:05:25.7273850Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7274125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7274240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7274522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7274649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7274854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7275131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7275251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7275536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7275663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7275833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7276187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7276489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7276652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7276805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7276865Z ok (3.736s) 2022-11-23T02:05:25.7276908Z 2022-11-23T02:05:25.7277107Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7277176Z Ran 1 test in 3.736s 2022-11-23T02:05:25.7277181Z 2022-11-23T02:05:25.7277233Z OK 2022-11-23T02:05:25.7277238Z 2022-11-23T02:05:25.7277314Z Generating XML reports... 2022-11-23T02:05:25.7277654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015944.xml 2022-11-23T02:05:25.7277888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7278163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7278285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7278571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7278702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7278707Z 2022-11-23T02:05:25.7278773Z Running tests... 2022-11-23T02:05:25.7278969Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7279226Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33211 2022-11-23T02:05:25.7279378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33212 2022-11-23T02:05:25.7279568Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7279844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7279963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7280251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7280379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7280550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7280819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7280936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7281220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7281385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7281554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7281862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7282164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7282324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7282482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7282542Z ok (3.735s) 2022-11-23T02:05:25.7282547Z 2022-11-23T02:05:25.7282744Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7282812Z Ran 1 test in 3.736s 2022-11-23T02:05:25.7282819Z 2022-11-23T02:05:25.7282874Z OK 2022-11-23T02:05:25.7282878Z 2022-11-23T02:05:25.7282954Z Generating XML reports... 2022-11-23T02:05:25.7283294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015951.xml 2022-11-23T02:05:25.7283557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7283839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7283955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7284240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7284368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7284373Z 2022-11-23T02:05:25.7284438Z Running tests... 2022-11-23T02:05:25.7284633Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7284829Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:05:25.7284839Z 2022-11-23T02:05:25.7285027Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7285097Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7285102Z 2022-11-23T02:05:25.7285165Z OK (skipped=1) 2022-11-23T02:05:25.7285169Z 2022-11-23T02:05:25.7285247Z Generating XML reports... 2022-11-23T02:05:25.7285585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015958.xml 2022-11-23T02:05:25.7285818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7286095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7286213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7286501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7286632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7286637Z 2022-11-23T02:05:25.7286701Z Running tests... 2022-11-23T02:05:25.7286896Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7287110Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.001s) 2022-11-23T02:05:25.7287116Z 2022-11-23T02:05:25.7287304Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7287371Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7287376Z 2022-11-23T02:05:25.7287439Z OK (skipped=1) 2022-11-23T02:05:25.7287443Z 2022-11-23T02:05:25.7287520Z Generating XML reports... 2022-11-23T02:05:25.7287859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020001.xml 2022-11-23T02:05:25.7288142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7288421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7288540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7288823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7288950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7288955Z 2022-11-23T02:05:25.7289015Z Running tests... 2022-11-23T02:05:25.7289208Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7289925Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:05:25.7289938Z 2022-11-23T02:05:25.7290161Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7290231Z Ran 1 test in 0.510s 2022-11-23T02:05:25.7290235Z 2022-11-23T02:05:25.7290299Z OK (skipped=1) 2022-11-23T02:05:25.7290303Z 2022-11-23T02:05:25.7290382Z Generating XML reports... 2022-11-23T02:05:25.7290724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020004.xml 2022-11-23T02:05:25.7290958Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7291234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7291354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7291637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7291765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7291772Z 2022-11-23T02:05:25.7291838Z Running tests... 2022-11-23T02:05:25.7292030Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7292272Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33568 2022-11-23T02:05:25.7292425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33569 2022-11-23T02:05:25.7292615Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7292891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7293008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7293291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7293420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7293592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7293896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7294173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7294289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7294569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7294732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7294903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7295209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7295366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7295525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7295704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzem8m5yw 2022-11-23T02:05:25.7295893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzem8m5yw/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7296071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe8pdg9f 2022-11-23T02:05:25.7296263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe8pdg9f/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7296323Z ok (6.349s) 2022-11-23T02:05:25.7296328Z 2022-11-23T02:05:25.7296559Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7296628Z Ran 1 test in 6.349s 2022-11-23T02:05:25.7296633Z 2022-11-23T02:05:25.7296687Z OK 2022-11-23T02:05:25.7296692Z 2022-11-23T02:05:25.7296768Z Generating XML reports... 2022-11-23T02:05:25.7297111Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020007.xml 2022-11-23T02:05:25.7297344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7297624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7297743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7298027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7298155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7298161Z 2022-11-23T02:05:25.7298227Z Running tests... 2022-11-23T02:05:25.7298419Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7298653Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33761 2022-11-23T02:05:25.7298805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33762 2022-11-23T02:05:25.7298993Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7299266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7299386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7299669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7299799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7299968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7300245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7300360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7300644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7300771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7300977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7301280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7301581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7301737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7301895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7301956Z ok (4.338s) 2022-11-23T02:05:25.7301961Z 2022-11-23T02:05:25.7302158Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7302226Z Ran 1 test in 4.339s 2022-11-23T02:05:25.7302232Z 2022-11-23T02:05:25.7302284Z OK 2022-11-23T02:05:25.7302288Z 2022-11-23T02:05:25.7302360Z Generating XML reports... 2022-11-23T02:05:25.7302705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020016.xml 2022-11-23T02:05:25.7302938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7303249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7303367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7303652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7303781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7303786Z 2022-11-23T02:05:25.7303852Z Running tests... 2022-11-23T02:05:25.7304049Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7304295Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33948 2022-11-23T02:05:25.7304449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33949 2022-11-23T02:05:25.7304640Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7304918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7305036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7305322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7305450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7305623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7305926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7306203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7306320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7306603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7306731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7306901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7307198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7307357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7307566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7307626Z ok (4.438s) 2022-11-23T02:05:25.7307631Z 2022-11-23T02:05:25.7307829Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7307897Z Ran 1 test in 4.439s 2022-11-23T02:05:25.7307902Z 2022-11-23T02:05:25.7307956Z OK 2022-11-23T02:05:25.7307960Z 2022-11-23T02:05:25.7308037Z Generating XML reports... 2022-11-23T02:05:25.7308382Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020024.xml 2022-11-23T02:05:25.7308616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7308892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7309012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7309296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7309426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7309431Z 2022-11-23T02:05:25.7309495Z Running tests... 2022-11-23T02:05:25.7309723Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7310437Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.513s) 2022-11-23T02:05:25.7310443Z 2022-11-23T02:05:25.7310635Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7310703Z Ran 1 test in 0.513s 2022-11-23T02:05:25.7310708Z 2022-11-23T02:05:25.7310773Z OK (skipped=1) 2022-11-23T02:05:25.7310778Z 2022-11-23T02:05:25.7310854Z Generating XML reports... 2022-11-23T02:05:25.7311193Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020031.xml 2022-11-23T02:05:25.7311428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7311706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7311823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7312106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7312235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7312240Z 2022-11-23T02:05:25.7312306Z Running tests... 2022-11-23T02:05:25.7312501Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7313231Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.505s) 2022-11-23T02:05:25.7313238Z 2022-11-23T02:05:25.7313428Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7313495Z Ran 1 test in 0.505s 2022-11-23T02:05:25.7313500Z 2022-11-23T02:05:25.7313562Z OK (skipped=1) 2022-11-23T02:05:25.7313566Z 2022-11-23T02:05:25.7313643Z Generating XML reports... 2022-11-23T02:05:25.7313984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020035.xml 2022-11-23T02:05:25.7314215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7314529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7314647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7314932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7315060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7315065Z 2022-11-23T02:05:25.7315129Z Running tests... 2022-11-23T02:05:25.7315324Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7316115Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75052 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.626s) 2022-11-23T02:05:25.7316124Z 2022-11-23T02:05:25.7316318Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7316386Z Ran 1 test in 0.626s 2022-11-23T02:05:25.7316391Z 2022-11-23T02:05:25.7316490Z OK (skipped=1) 2022-11-23T02:05:25.7316496Z 2022-11-23T02:05:25.7316572Z Generating XML reports... 2022-11-23T02:05:25.7316914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020038.xml 2022-11-23T02:05:25.7317146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7317424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7317542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7317825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7317955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7317959Z 2022-11-23T02:05:25.7318023Z Running tests... 2022-11-23T02:05:25.7318215Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7318976Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75139 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:05:25.7318987Z 2022-11-23T02:05:25.7319178Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7319241Z Ran 1 test in 0.510s 2022-11-23T02:05:25.7319251Z 2022-11-23T02:05:25.7319310Z OK (skipped=1) 2022-11-23T02:05:25.7319314Z 2022-11-23T02:05:25.7319391Z Generating XML reports... 2022-11-23T02:05:25.7319728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020042.xml 2022-11-23T02:05:25.7319962Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7320237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7320356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7320638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7320766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7320770Z 2022-11-23T02:05:25.7320834Z Running tests... 2022-11-23T02:05:25.7321026Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7321780Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.511s) 2022-11-23T02:05:25.7321786Z 2022-11-23T02:05:25.7321977Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7322045Z Ran 1 test in 0.511s 2022-11-23T02:05:25.7322050Z 2022-11-23T02:05:25.7322112Z OK (skipped=1) 2022-11-23T02:05:25.7322117Z 2022-11-23T02:05:25.7322194Z Generating XML reports... 2022-11-23T02:05:25.7322531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020046.xml 2022-11-23T02:05:25.7322763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7323041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7323157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7323472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7323601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7323606Z 2022-11-23T02:05:25.7323670Z Running tests... 2022-11-23T02:05:25.7323866Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7324096Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34425 2022-11-23T02:05:25.7324245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34426 2022-11-23T02:05:25.7324436Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7324712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7324829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7325110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7325237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7325406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7325709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7325984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7326104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7326388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7326518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7326689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7326992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7327153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7327309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7327478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7327683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7327983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7328280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7328538Z STAGE:2022-11-23 02:00:52 34425:34425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7328782Z STAGE:2022-11-23 02:00:52 34426:34426 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7329033Z STAGE:2022-11-23 02:00:52 34426:34426 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7329297Z STAGE:2022-11-23 02:00:52 34426:34426 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7329549Z STAGE:2022-11-23 02:00:52 34425:34425 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7329806Z STAGE:2022-11-23 02:00:52 34425:34425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7330050Z STAGE:2022-11-23 02:00:52 34426:34426 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7330323Z STAGE:2022-11-23 02:00:52 34425:34425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7330575Z STAGE:2022-11-23 02:00:52 34426:34426 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7330834Z STAGE:2022-11-23 02:00:52 34426:34426 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7331077Z STAGE:2022-11-23 02:00:52 34425:34425 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7331334Z STAGE:2022-11-23 02:00:52 34425:34425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7331396Z ok (4.136s) 2022-11-23T02:05:25.7331401Z 2022-11-23T02:05:25.7331600Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7331669Z Ran 1 test in 4.136s 2022-11-23T02:05:25.7331674Z 2022-11-23T02:05:25.7331727Z OK 2022-11-23T02:05:25.7331731Z 2022-11-23T02:05:25.7331808Z Generating XML reports... 2022-11-23T02:05:25.7332152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020049.xml 2022-11-23T02:05:25.7332387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7332664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7332781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7333064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7333194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7333199Z 2022-11-23T02:05:25.7333265Z Running tests... 2022-11-23T02:05:25.7333461Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7333691Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34620 2022-11-23T02:05:25.7333843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34621 2022-11-23T02:05:25.7334028Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7334302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7334420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7334703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7334867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7335037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7335316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7335431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7335714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7335842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7336011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7336314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7336614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7336774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7336966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7337139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7337305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7337605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7337858Z STAGE:2022-11-23 02:00:59 34620:34620 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7338150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7338402Z STAGE:2022-11-23 02:00:59 34621:34621 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7338654Z STAGE:2022-11-23 02:00:59 34620:34620 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7338902Z STAGE:2022-11-23 02:00:59 34621:34621 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7339164Z STAGE:2022-11-23 02:00:59 34620:34620 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7339421Z STAGE:2022-11-23 02:00:59 34621:34621 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7339668Z STAGE:2022-11-23 02:00:59 34621:34621 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7339907Z STAGE:2022-11-23 02:00:59 34620:34620 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7340332Z STAGE:2022-11-23 02:00:59 34620:34620 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:00:59 34621:34621 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7340338Z 2022-11-23T02:05:25.7340779Z STAGE:2022-11-23 02:00:59 34621:34621 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:00:59 34620:34620 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7340789Z 2022-11-23T02:05:25.7340846Z ok (4.148s) 2022-11-23T02:05:25.7340850Z 2022-11-23T02:05:25.7341051Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7341118Z Ran 1 test in 4.149s 2022-11-23T02:05:25.7341123Z 2022-11-23T02:05:25.7341177Z OK 2022-11-23T02:05:25.7341181Z 2022-11-23T02:05:25.7341258Z Generating XML reports... 2022-11-23T02:05:25.7341600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020056.xml 2022-11-23T02:05:25.7341867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7342145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7342267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7342553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7342681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7342686Z 2022-11-23T02:05:25.7342752Z Running tests... 2022-11-23T02:05:25.7342948Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7343183Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34815 2022-11-23T02:05:25.7343334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34816 2022-11-23T02:05:25.7343529Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7343805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7343955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7344242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7344370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7344545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7344823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7344936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7345220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7345347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7345519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7345821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7346121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7346279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7346436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7346604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7346772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7347068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7347366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7347620Z STAGE:2022-11-23 02:01:06 34815:34815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7347872Z STAGE:2022-11-23 02:01:06 34816:34816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7348126Z STAGE:2022-11-23 02:01:06 34816:34816 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7348390Z STAGE:2022-11-23 02:01:06 34816:34816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7348676Z STAGE:2022-11-23 02:01:06 34815:34815 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7348938Z STAGE:2022-11-23 02:01:06 34815:34815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7349185Z STAGE:2022-11-23 02:01:06 34816:34816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7349426Z STAGE:2022-11-23 02:01:06 34815:34815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7349676Z STAGE:2022-11-23 02:01:06 34816:34816 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7349934Z STAGE:2022-11-23 02:01:06 34816:34816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7350181Z STAGE:2022-11-23 02:01:06 34815:34815 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7350438Z STAGE:2022-11-23 02:01:06 34815:34815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7350496Z ok (3.951s) 2022-11-23T02:05:25.7350505Z 2022-11-23T02:05:25.7350698Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7350767Z Ran 1 test in 3.952s 2022-11-23T02:05:25.7350772Z 2022-11-23T02:05:25.7350825Z OK 2022-11-23T02:05:25.7350829Z 2022-11-23T02:05:25.7350937Z Generating XML reports... 2022-11-23T02:05:25.7351283Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020103.xml 2022-11-23T02:05:25.7351519Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7351798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7351917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7352200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7352332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7352337Z 2022-11-23T02:05:25.7352401Z Running tests... 2022-11-23T02:05:25.7352599Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7352833Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35010 2022-11-23T02:05:25.7352984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35011 2022-11-23T02:05:25.7353173Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7353451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7353567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7353856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7353984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7354155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7354439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7354551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7354833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7354961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7355130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7355434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7355767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7355929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7356132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7356302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7356467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7356767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7357061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7357318Z STAGE:2022-11-23 02:01:13 35010:35010 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7357567Z STAGE:2022-11-23 02:01:13 35011:35011 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7357871Z STAGE:2022-11-23 02:01:13 35011:35011 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7358123Z STAGE:2022-11-23 02:01:13 35010:35010 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7358386Z STAGE:2022-11-23 02:01:13 35011:35011 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7358648Z STAGE:2022-11-23 02:01:13 35010:35010 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7358890Z STAGE:2022-11-23 02:01:13 35011:35011 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7359130Z STAGE:2022-11-23 02:01:13 35010:35010 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7359380Z STAGE:2022-11-23 02:01:13 35011:35011 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7359637Z STAGE:2022-11-23 02:01:13 35011:35011 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7359891Z STAGE:2022-11-23 02:01:13 35010:35010 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7360152Z STAGE:2022-11-23 02:01:13 35010:35010 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7360207Z ok (3.941s) 2022-11-23T02:05:25.7360218Z 2022-11-23T02:05:25.7360410Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7360479Z Ran 1 test in 3.942s 2022-11-23T02:05:25.7360484Z 2022-11-23T02:05:25.7360536Z OK 2022-11-23T02:05:25.7360541Z 2022-11-23T02:05:25.7360618Z Generating XML reports... 2022-11-23T02:05:25.7360961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020110.xml 2022-11-23T02:05:25.7361198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7361477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7361596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7361879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7362007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7362012Z 2022-11-23T02:05:25.7362078Z Running tests... 2022-11-23T02:05:25.7362274Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7362500Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35205 2022-11-23T02:05:25.7362689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35206 2022-11-23T02:05:25.7362879Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7363159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7363276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7363559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7363686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7363856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7364158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7364434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7364547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7364884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7365012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7365183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7365483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7365642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7365799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7365908Z skip: Skipped due to small world size. (3.741s) 2022-11-23T02:05:25.7365913Z 2022-11-23T02:05:25.7366107Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7366175Z Ran 1 test in 3.742s 2022-11-23T02:05:25.7366180Z 2022-11-23T02:05:25.7366243Z OK (skipped=1) 2022-11-23T02:05:25.7366249Z 2022-11-23T02:05:25.7366326Z Generating XML reports... 2022-11-23T02:05:25.7366670Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020117.xml 2022-11-23T02:05:25.7366907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7367185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7367302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7367584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7367715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7367720Z 2022-11-23T02:05:25.7367785Z Running tests... 2022-11-23T02:05:25.7367984Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7368210Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35388 2022-11-23T02:05:25.7368361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35389 2022-11-23T02:05:25.7368548Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7368822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7368938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7369261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7369391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7369562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7369838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7369953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7370232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7370359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7370529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7370831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7371130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7371321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7371479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7371582Z skip: Skipped due to small world size. (3.740s) 2022-11-23T02:05:25.7371587Z 2022-11-23T02:05:25.7371784Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7371852Z Ran 1 test in 3.741s 2022-11-23T02:05:25.7371857Z 2022-11-23T02:05:25.7371919Z OK (skipped=1) 2022-11-23T02:05:25.7371924Z 2022-11-23T02:05:25.7372000Z Generating XML reports... 2022-11-23T02:05:25.7372344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020124.xml 2022-11-23T02:05:25.7372579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7372857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7372972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7373255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7373384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7373388Z 2022-11-23T02:05:25.7373452Z Running tests... 2022-11-23T02:05:25.7373644Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7373877Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35571 2022-11-23T02:05:25.7374030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35572 2022-11-23T02:05:25.7374221Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7374497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7374612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7374896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7375023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7375192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7375466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7375614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7375900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7376027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7376195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7376500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7376797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7376955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7377110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7377216Z skip: Skipped due to small world size. (3.741s) 2022-11-23T02:05:25.7377221Z 2022-11-23T02:05:25.7377412Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7377482Z Ran 1 test in 3.741s 2022-11-23T02:05:25.7377487Z 2022-11-23T02:05:25.7377550Z OK (skipped=1) 2022-11-23T02:05:25.7377554Z 2022-11-23T02:05:25.7377662Z Generating XML reports... 2022-11-23T02:05:25.7378007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020131.xml 2022-11-23T02:05:25.7378239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7378515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7378632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7378917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7379048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7379053Z 2022-11-23T02:05:25.7379117Z Running tests... 2022-11-23T02:05:25.7379310Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7379540Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35754 2022-11-23T02:05:25.7379690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35755 2022-11-23T02:05:25.7379883Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7380157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7380275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7380557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7380685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7380855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7381133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7381251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7381531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7381659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7381831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7382131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7382550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7382711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7382871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7382979Z skip: Skipped due to small world size. (3.638s) 2022-11-23T02:05:25.7382984Z 2022-11-23T02:05:25.7383182Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7383249Z Ran 1 test in 3.639s 2022-11-23T02:05:25.7383254Z 2022-11-23T02:05:25.7387131Z OK (skipped=1) 2022-11-23T02:05:25.7387139Z 2022-11-23T02:05:25.7387231Z Generating XML reports... 2022-11-23T02:05:25.7387610Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020138.xml 2022-11-23T02:05:25.7387854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7388139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7388328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7388619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7388747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7388754Z 2022-11-23T02:05:25.7388818Z Running tests... 2022-11-23T02:05:25.7389020Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7389244Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35937 2022-11-23T02:05:25.7389398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35938 2022-11-23T02:05:25.7389590Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7389877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7389996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7390283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7390409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7390584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7390863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7390982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7391265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7391391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7391563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7391869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7392169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7392327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7392584Z STAGE:2022-11-23 02:01:47 35938:35938 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7392781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7393036Z STAGE:2022-11-23 02:01:47 35937:35937 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7393291Z STAGE:2022-11-23 02:01:47 35937:35937 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7393558Z STAGE:2022-11-23 02:01:47 35937:35937 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7393810Z STAGE:2022-11-23 02:01:47 35938:35938 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7394070Z STAGE:2022-11-23 02:01:47 35938:35938 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7394313Z STAGE:2022-11-23 02:01:47 35938:35938 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7394555Z STAGE:2022-11-23 02:01:47 35937:35937 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7394806Z STAGE:2022-11-23 02:01:47 35937:35937 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7395051Z STAGE:2022-11-23 02:01:47 35938:35938 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7395341Z STAGE:2022-11-23 02:01:47 35937:35937 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7395601Z STAGE:2022-11-23 02:01:47 35938:35938 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7395656Z ok (3.951s) 2022-11-23T02:05:25.7395665Z 2022-11-23T02:05:25.7395858Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7395927Z Ran 1 test in 3.952s 2022-11-23T02:05:25.7395931Z 2022-11-23T02:05:25.7395983Z OK 2022-11-23T02:05:25.7395987Z 2022-11-23T02:05:25.7396194Z Generating XML reports... 2022-11-23T02:05:25.7396761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020145.xml 2022-11-23T02:05:25.7397013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7397309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7397434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7397735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7397870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7397876Z 2022-11-23T02:05:25.7397944Z Running tests... 2022-11-23T02:05:25.7398150Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7398386Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36126 2022-11-23T02:05:25.7398548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36127 2022-11-23T02:05:25.7398755Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7399051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7399177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7399486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7399621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7399798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7400121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7400413Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7400592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7400892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7401029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7401208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7401524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7401692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7401961Z STAGE:2022-11-23 02:01:54 36127:36127 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7402129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7402396Z STAGE:2022-11-23 02:01:54 36126:36126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7402665Z STAGE:2022-11-23 02:01:54 36126:36126 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7403182Z STAGE:2022-11-23 02:01:54 36127:36127 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:01:54 36126:36126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7403190Z 2022-11-23T02:05:25.7403474Z STAGE:2022-11-23 02:01:54 36127:36127 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7403736Z STAGE:2022-11-23 02:01:54 36127:36127 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7403991Z STAGE:2022-11-23 02:01:54 36126:36126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7404255Z STAGE:2022-11-23 02:01:54 36126:36126 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7404533Z STAGE:2022-11-23 02:01:54 36126:36126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7404796Z STAGE:2022-11-23 02:01:54 36127:36127 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7405074Z STAGE:2022-11-23 02:01:54 36127:36127 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7405139Z ok (3.852s) 2022-11-23T02:05:25.7405145Z 2022-11-23T02:05:25.7405353Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7405425Z Ran 1 test in 3.852s 2022-11-23T02:05:25.7405431Z 2022-11-23T02:05:25.7405486Z OK 2022-11-23T02:05:25.7405491Z 2022-11-23T02:05:25.7405566Z Generating XML reports... 2022-11-23T02:05:25.7405933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020152.xml 2022-11-23T02:05:25.7406183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7406477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7406602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7406907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7407043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7407049Z 2022-11-23T02:05:25.7407117Z Running tests... 2022-11-23T02:05:25.7407328Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7407536Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s) 2022-11-23T02:05:25.7407541Z 2022-11-23T02:05:25.7407746Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7407868Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7407873Z 2022-11-23T02:05:25.7407940Z OK (skipped=1) 2022-11-23T02:05:25.7407945Z 2022-11-23T02:05:25.7408026Z Generating XML reports... 2022-11-23T02:05:25.7408391Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020159.xml 2022-11-23T02:05:25.7408640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7408936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7409060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7409360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7409495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7409503Z 2022-11-23T02:05:25.7409572Z Running tests... 2022-11-23T02:05:25.7409780Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7410021Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36373 2022-11-23T02:05:25.7410217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36374 2022-11-23T02:05:25.7410423Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7410718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7410844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7411146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7411282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7411466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7411784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7412082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7412206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7412508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7412745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7412915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7413216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7413379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7413634Z STAGE:2022-11-23 02:02:04 36374:36374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7413794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7414050Z STAGE:2022-11-23 02:02:04 36373:36373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7414304Z STAGE:2022-11-23 02:02:04 36374:36374 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7414567Z STAGE:2022-11-23 02:02:04 36374:36374 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7414981Z STAGE:2022-11-23 02:02:04 36373:36373 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7415259Z STAGE:2022-11-23 02:02:04 36373:36373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7415585Z STAGE:2022-11-23 02:02:04 36374:36374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7415835Z STAGE:2022-11-23 02:02:04 36373:36373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7416101Z STAGE:2022-11-23 02:02:04 36373:36373 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7416377Z STAGE:2022-11-23 02:02:04 36373:36373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7416642Z STAGE:2022-11-23 02:02:04 36374:36374 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7416914Z STAGE:2022-11-23 02:02:04 36374:36374 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7416978Z ok (4.037s) 2022-11-23T02:05:25.7416984Z 2022-11-23T02:05:25.7417192Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7417267Z Ran 1 test in 4.038s 2022-11-23T02:05:25.7417272Z 2022-11-23T02:05:25.7417328Z OK 2022-11-23T02:05:25.7417332Z 2022-11-23T02:05:25.7417413Z Generating XML reports... 2022-11-23T02:05:25.7417772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020201.xml 2022-11-23T02:05:25.7418054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7418352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7418477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7418780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7418915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7418921Z 2022-11-23T02:05:25.7418992Z Running tests... 2022-11-23T02:05:25.7419199Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7419424Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s) 2022-11-23T02:05:25.7419430Z 2022-11-23T02:05:25.7419636Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7419707Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7419712Z 2022-11-23T02:05:25.7419778Z OK (skipped=1) 2022-11-23T02:05:25.7419783Z 2022-11-23T02:05:25.7419863Z Generating XML reports... 2022-11-23T02:05:25.7420221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020208.xml 2022-11-23T02:05:25.7420469Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7420761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7420888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7421187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7421324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7421331Z 2022-11-23T02:05:25.7421399Z Running tests... 2022-11-23T02:05:25.7421605Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7421814Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.002s) 2022-11-23T02:05:25.7421819Z 2022-11-23T02:05:25.7422023Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7422096Z Ran 1 test in 0.003s 2022-11-23T02:05:25.7422101Z 2022-11-23T02:05:25.7422168Z OK (skipped=1) 2022-11-23T02:05:25.7422173Z 2022-11-23T02:05:25.7422253Z Generating XML reports... 2022-11-23T02:05:25.7422660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020211.xml 2022-11-23T02:05:25.7422905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7423199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7423326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7423627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7423763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7423769Z 2022-11-23T02:05:25.7423836Z Running tests... 2022-11-23T02:05:25.7424042Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7424273Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36678 2022-11-23T02:05:25.7424432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36679 2022-11-23T02:05:25.7424631Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7424979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7425104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7425405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7425540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7425723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7426045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7426338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7426461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7426764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7426901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7427081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7427400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7427569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7427847Z STAGE:2022-11-23 02:02:17 36679:36679 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7428015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7428284Z STAGE:2022-11-23 02:02:17 36678:36678 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7428558Z STAGE:2022-11-23 02:02:17 36678:36678 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7428835Z STAGE:2022-11-23 02:02:17 36678:36678 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7429100Z STAGE:2022-11-23 02:02:17 36679:36679 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7429375Z STAGE:2022-11-23 02:02:17 36679:36679 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7429633Z STAGE:2022-11-23 02:02:17 36678:36678 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7429922Z STAGE:2022-11-23 02:02:17 36679:36679 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7430179Z STAGE:2022-11-23 02:02:17 36678:36678 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7430452Z STAGE:2022-11-23 02:02:17 36678:36678 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7430815Z STAGE:2022-11-23 02:02:17 36679:36679 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7431076Z STAGE:2022-11-23 02:02:17 36679:36679 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7431136Z ok (3.938s) 2022-11-23T02:05:25.7431141Z 2022-11-23T02:05:25.7431336Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7431404Z Ran 1 test in 3.939s 2022-11-23T02:05:25.7431409Z 2022-11-23T02:05:25.7431461Z OK 2022-11-23T02:05:25.7431465Z 2022-11-23T02:05:25.7431542Z Generating XML reports... 2022-11-23T02:05:25.7432048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020214.xml 2022-11-23T02:05:25.7432297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7432624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7432749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7433054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7433189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7433195Z 2022-11-23T02:05:25.7433263Z Running tests... 2022-11-23T02:05:25.7433471Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7433665Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-11-23T02:05:25.7433673Z 2022-11-23T02:05:25.7433876Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7433947Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7433952Z 2022-11-23T02:05:25.7434021Z OK (skipped=1) 2022-11-23T02:05:25.7434027Z 2022-11-23T02:05:25.7434110Z Generating XML reports... 2022-11-23T02:05:25.7434462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020221.xml 2022-11-23T02:05:25.7434707Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7435002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7435127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7435431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7435570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7435575Z 2022-11-23T02:05:25.7435643Z Running tests... 2022-11-23T02:05:25.7435850Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7436101Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-11-23T02:05:25.7436106Z 2022-11-23T02:05:25.7436311Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7436383Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7436388Z 2022-11-23T02:05:25.7436455Z OK (skipped=1) 2022-11-23T02:05:25.7436460Z 2022-11-23T02:05:25.7436542Z Generating XML reports... 2022-11-23T02:05:25.7436901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020224.xml 2022-11-23T02:05:25.7437148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7437491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7437615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7437916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7438054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7438059Z 2022-11-23T02:05:25.7438129Z Running tests... 2022-11-23T02:05:25.7438337Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7438576Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36983 2022-11-23T02:05:25.7438735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36984 2022-11-23T02:05:25.7438932Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7439226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7439351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7439695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7439832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7440014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7440307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7440431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7440728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7440865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7441046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7441372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7441687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7441855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7442123Z STAGE:2022-11-23 02:02:30 36984:36984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7442289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7442556Z STAGE:2022-11-23 02:02:30 36983:36983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7443007Z STAGE:2022-11-23 02:02:30 36983:36983 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:02:30 36984:36984 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7443013Z 2022-11-23T02:05:25.7443485Z STAGE:2022-11-23 02:02:30 36983:36983 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:02:30 36984:36984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7443491Z 2022-11-23T02:05:25.7443752Z STAGE:2022-11-23 02:02:30 36984:36984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7444009Z STAGE:2022-11-23 02:02:30 36983:36983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7444277Z STAGE:2022-11-23 02:02:30 36983:36983 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7444595Z STAGE:2022-11-23 02:02:30 36983:36983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7444861Z STAGE:2022-11-23 02:02:30 36984:36984 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7445138Z STAGE:2022-11-23 02:02:30 36984:36984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7445203Z ok (3.938s) 2022-11-23T02:05:25.7445208Z 2022-11-23T02:05:25.7445417Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7445485Z Ran 1 test in 3.939s 2022-11-23T02:05:25.7445494Z 2022-11-23T02:05:25.7445545Z OK 2022-11-23T02:05:25.7445550Z 2022-11-23T02:05:25.7445631Z Generating XML reports... 2022-11-23T02:05:25.7445992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020227.xml 2022-11-23T02:05:25.7446240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7446536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7446663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7446999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7447139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7447144Z 2022-11-23T02:05:25.7447212Z Running tests... 2022-11-23T02:05:25.7447421Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7447748Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37172 2022-11-23T02:05:25.7447900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37173 2022-11-23T02:05:25.7448091Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7448370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7448488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7448775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7448903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7449075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7449350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7449467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7449753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7449879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7450051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7450356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7450655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7450813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7451064Z STAGE:2022-11-23 02:02:36 37173:37173 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7451221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7451473Z STAGE:2022-11-23 02:02:37 37172:37172 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7451763Z STAGE:2022-11-23 02:02:37 37172:37172 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7452028Z STAGE:2022-11-23 02:02:37 37172:37172 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7452282Z STAGE:2022-11-23 02:02:37 37173:37173 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7452543Z STAGE:2022-11-23 02:02:37 37173:37173 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7452789Z STAGE:2022-11-23 02:02:37 37172:37172 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7453029Z STAGE:2022-11-23 02:02:37 37173:37173 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7453278Z STAGE:2022-11-23 02:02:37 37173:37173 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7453536Z STAGE:2022-11-23 02:02:37 37173:37173 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7453789Z STAGE:2022-11-23 02:02:37 37172:37172 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7454047Z STAGE:2022-11-23 02:02:37 37172:37172 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7454141Z ok (3.837s) 2022-11-23T02:05:25.7454147Z 2022-11-23T02:05:25.7454348Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7454415Z Ran 1 test in 3.838s 2022-11-23T02:05:25.7454420Z 2022-11-23T02:05:25.7454472Z OK 2022-11-23T02:05:25.7454476Z 2022-11-23T02:05:25.7454553Z Generating XML reports... 2022-11-23T02:05:25.7454892Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020234.xml 2022-11-23T02:05:25.7455128Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7455409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7455527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7455810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7455943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7455948Z 2022-11-23T02:05:25.7456013Z Running tests... 2022-11-23T02:05:25.7456209Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7456436Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37361 2022-11-23T02:05:25.7456589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37362 2022-11-23T02:05:25.7456778Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7457059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7457177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7457466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7457596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7457769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7458073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7458350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7458467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7458787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7458916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7459088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7459387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7459542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7459700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7459761Z ok (3.841s) 2022-11-23T02:05:25.7459766Z 2022-11-23T02:05:25.7459966Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7460033Z Ran 1 test in 3.842s 2022-11-23T02:05:25.7460040Z 2022-11-23T02:05:25.7460095Z OK 2022-11-23T02:05:25.7460099Z 2022-11-23T02:05:25.7460174Z Generating XML reports... 2022-11-23T02:05:25.7460519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020241.xml 2022-11-23T02:05:25.7460787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7461067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7461184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7461474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7461602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7461607Z 2022-11-23T02:05:25.7461673Z Running tests... 2022-11-23T02:05:25.7461870Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7462101Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37544 2022-11-23T02:05:25.7462254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37545 2022-11-23T02:05:25.7462446Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7462721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7462837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7463127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7463255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7463421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7463726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7464003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7464120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7464406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7464533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7464703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7465007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7465203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7465461Z STAGE:2022-11-23 02:02:50 37545:37545 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7465620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7465874Z STAGE:2022-11-23 02:02:50 37544:37544 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7466128Z STAGE:2022-11-23 02:02:50 37544:37544 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7466392Z STAGE:2022-11-23 02:02:50 37544:37544 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7466642Z STAGE:2022-11-23 02:02:50 37545:37545 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7466901Z STAGE:2022-11-23 02:02:50 37545:37545 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7467148Z STAGE:2022-11-23 02:02:50 37544:37544 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7467392Z STAGE:2022-11-23 02:02:50 37545:37545 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7467686Z STAGE:2022-11-23 02:02:50 37544:37544 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7467933Z STAGE:2022-11-23 02:02:50 37545:37545 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7468191Z STAGE:2022-11-23 02:02:50 37544:37544 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7468450Z STAGE:2022-11-23 02:02:50 37545:37545 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7468509Z ok (3.838s) 2022-11-23T02:05:25.7468514Z 2022-11-23T02:05:25.7468705Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7468773Z Ran 1 test in 3.839s 2022-11-23T02:05:25.7468781Z 2022-11-23T02:05:25.7468833Z OK 2022-11-23T02:05:25.7468837Z 2022-11-23T02:05:25.7468912Z Generating XML reports... 2022-11-23T02:05:25.7469256Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020248.xml 2022-11-23T02:05:25.7469488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7469767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7469884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7470167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7470297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7470302Z 2022-11-23T02:05:25.7470369Z Running tests... 2022-11-23T02:05:25.7470565Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7470751Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s) 2022-11-23T02:05:25.7470756Z 2022-11-23T02:05:25.7470948Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7471016Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7471023Z 2022-11-23T02:05:25.7471087Z OK (skipped=1) 2022-11-23T02:05:25.7471092Z 2022-11-23T02:05:25.7471170Z Generating XML reports... 2022-11-23T02:05:25.7471508Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020255.xml 2022-11-23T02:05:25.7471740Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7472015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7472133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7472453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7472575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7472586Z 2022-11-23T02:05:25.7472645Z Running tests... 2022-11-23T02:05:25.7472840Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7473029Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s) 2022-11-23T02:05:25.7473034Z 2022-11-23T02:05:25.7473230Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7473298Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7473303Z 2022-11-23T02:05:25.7473365Z OK (skipped=1) 2022-11-23T02:05:25.7473370Z 2022-11-23T02:05:25.7473447Z Generating XML reports... 2022-11-23T02:05:25.7473785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020258.xml 2022-11-23T02:05:25.7474019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7474295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7474447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7474733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7474861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7474867Z 2022-11-23T02:05:25.7474932Z Running tests... 2022-11-23T02:05:25.7475124Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7475354Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37849 2022-11-23T02:05:25.7475507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37850 2022-11-23T02:05:25.7475697Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7475975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7476134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7476593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7476729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7476907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7477229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7477524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7477648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7477954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7478090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7478268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7478590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7478760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7478929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7479153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:05:25.7479334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:05:25.7479652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7479965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:05:25.7480235Z STAGE:2022-11-23 02:03:03 37849:37849 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7480498Z STAGE:2022-11-23 02:03:03 37850:37850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7480767Z STAGE:2022-11-23 02:03:03 37850:37850 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7481029Z STAGE:2022-11-23 02:03:03 37849:37849 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7481312Z STAGE:2022-11-23 02:03:03 37850:37850 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7481589Z STAGE:2022-11-23 02:03:03 37849:37849 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7481884Z STAGE:2022-11-23 02:03:03 37849:37849 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7482140Z STAGE:2022-11-23 02:03:03 37850:37850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7482400Z STAGE:2022-11-23 02:03:03 37850:37850 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7482673Z STAGE:2022-11-23 02:03:03 37850:37850 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7482932Z STAGE:2022-11-23 02:03:03 37849:37849 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7483203Z STAGE:2022-11-23 02:03:03 37849:37849 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7483270Z ok (3.849s) 2022-11-23T02:05:25.7483275Z 2022-11-23T02:05:25.7483483Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7483554Z Ran 1 test in 3.850s 2022-11-23T02:05:25.7483560Z 2022-11-23T02:05:25.7483615Z OK 2022-11-23T02:05:25.7483622Z 2022-11-23T02:05:25.7483705Z Generating XML reports... 2022-11-23T02:05:25.7484071Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020301.xml 2022-11-23T02:05:25.7484317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7484608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7484732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7485032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7485169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7485175Z 2022-11-23T02:05:25.7485243Z Running tests... 2022-11-23T02:05:25.7485450Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7485688Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38044 2022-11-23T02:05:25.7485849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38045 2022-11-23T02:05:25.7486052Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7486344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7486467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7486803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7487048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7487216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7487495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7487612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7487896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7488024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7488194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7488494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7488797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7488991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7489156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7489262Z skip: Skipped due to small world size. (3.839s) 2022-11-23T02:05:25.7489267Z 2022-11-23T02:05:25.7489467Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7489535Z Ran 1 test in 3.840s 2022-11-23T02:05:25.7489540Z 2022-11-23T02:05:25.7489605Z OK (skipped=1) 2022-11-23T02:05:25.7489610Z 2022-11-23T02:05:25.7489687Z Generating XML reports... 2022-11-23T02:05:25.7490030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020307.xml 2022-11-23T02:05:25.7490267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7490545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7490664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7490949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7491077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7491082Z 2022-11-23T02:05:25.7491149Z Running tests... 2022-11-23T02:05:25.7491346Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7491573Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38227 2022-11-23T02:05:25.7491725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38228 2022-11-23T02:05:25.7491916Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7492194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7492311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7492594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7492723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7492897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7493173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7493325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7493610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7493738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7493912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7494215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7494519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7494681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7494837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7494898Z ok (3.851s) 2022-11-23T02:05:25.7494903Z 2022-11-23T02:05:25.7495101Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7495169Z Ran 1 test in 3.852s 2022-11-23T02:05:25.7495174Z 2022-11-23T02:05:25.7495227Z OK 2022-11-23T02:05:25.7495231Z 2022-11-23T02:05:25.7495308Z Generating XML reports... 2022-11-23T02:05:25.7495678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020314.xml 2022-11-23T02:05:25.7495912Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7496193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7496310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7496594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7496724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7496730Z 2022-11-23T02:05:25.7496796Z Running tests... 2022-11-23T02:05:25.7496993Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7497218Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38410 2022-11-23T02:05:25.7497371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38411 2022-11-23T02:05:25.7497561Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7497837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7497955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7498236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7498367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7498538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7498816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7498935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7499216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7499343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7499514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7499815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7500152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7500305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7500464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7500524Z ok (3.739s) 2022-11-23T02:05:25.7500529Z 2022-11-23T02:05:25.7500726Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7500794Z Ran 1 test in 3.739s 2022-11-23T02:05:25.7500799Z 2022-11-23T02:05:25.7500850Z OK 2022-11-23T02:05:25.7500855Z 2022-11-23T02:05:25.7500930Z Generating XML reports... 2022-11-23T02:05:25.7501277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020321.xml 2022-11-23T02:05:25.7501512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7501793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7501912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7502227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7502357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7502362Z 2022-11-23T02:05:25.7502427Z Running tests... 2022-11-23T02:05:25.7502625Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7502855Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38593 2022-11-23T02:05:25.7503007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38594 2022-11-23T02:05:25.7503201Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7503477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7503594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7503878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7504005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7504171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7504448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7504564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7504847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7504976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7505149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7505449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7505749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7505907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7506064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7506127Z ok (3.838s) 2022-11-23T02:05:25.7506132Z 2022-11-23T02:05:25.7506327Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7506429Z Ran 1 test in 3.838s 2022-11-23T02:05:25.7506434Z 2022-11-23T02:05:25.7506486Z OK 2022-11-23T02:05:25.7506491Z 2022-11-23T02:05:25.7506568Z Generating XML reports... 2022-11-23T02:05:25.7506915Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020328.xml 2022-11-23T02:05:25.7507149Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7507430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7507546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7507828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7507957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7507962Z 2022-11-23T02:05:25.7508030Z Running tests... 2022-11-23T02:05:25.7508223Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7508475Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38776 2022-11-23T02:05:25.7508656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38777 2022-11-23T02:05:25.7508847Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7509121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7509238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7509520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7509649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7509824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7510100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7510219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7510506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7510632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7510803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7511105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7511402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7511566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7511725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7511983Z STAGE:2022-11-23 02:03:37 38776:38776 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7512232Z STAGE:2022-11-23 02:03:37 38777:38777 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7512487Z STAGE:2022-11-23 02:03:37 38776:38776 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7512750Z STAGE:2022-11-23 02:03:37 38776:38776 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7513001Z STAGE:2022-11-23 02:03:37 38777:38777 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7513264Z STAGE:2022-11-23 02:03:37 38777:38777 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7513353Z ok (3.751s) 2022-11-23T02:05:25.7513358Z 2022-11-23T02:05:25.7513558Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7513626Z Ran 1 test in 3.751s 2022-11-23T02:05:25.7513631Z 2022-11-23T02:05:25.7513685Z OK 2022-11-23T02:05:25.7513690Z 2022-11-23T02:05:25.7513766Z Generating XML reports... 2022-11-23T02:05:25.7514109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020335.xml 2022-11-23T02:05:25.7514341Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7514618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7514737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7515023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7515152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7515158Z 2022-11-23T02:05:25.7515224Z Running tests... 2022-11-23T02:05:25.7515459Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7515707Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38965 2022-11-23T02:05:25.7515861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38966 2022-11-23T02:05:25.7516097Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7516378Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7516665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7516971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7517106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7517288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7517578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7517696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7517996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7518131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7518312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7518633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7518953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7519123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7519390Z STAGE:2022-11-23 02:03:44 38966:38966 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7519557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7519830Z STAGE:2022-11-23 02:03:44 38965:38965 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7520099Z STAGE:2022-11-23 02:03:44 38965:38965 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7520375Z STAGE:2022-11-23 02:03:44 38965:38965 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7520705Z STAGE:2022-11-23 02:03:44 38966:38966 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7520982Z STAGE:2022-11-23 02:03:44 38966:38966 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7521048Z ok (3.951s) 2022-11-23T02:05:25.7521056Z 2022-11-23T02:05:25.7521267Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7521340Z Ran 1 test in 3.952s 2022-11-23T02:05:25.7521345Z 2022-11-23T02:05:25.7521401Z OK 2022-11-23T02:05:25.7521405Z 2022-11-23T02:05:25.7521485Z Generating XML reports... 2022-11-23T02:05:25.7521848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020342.xml 2022-11-23T02:05:25.7522095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7522390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7522512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7522818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7522996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7523002Z 2022-11-23T02:05:25.7523071Z Running tests... 2022-11-23T02:05:25.7523281Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7523534Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39154 2022-11-23T02:05:25.7523693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39155 2022-11-23T02:05:25.7523892Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7524189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7524315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7524616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7524751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7524934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7525222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7525345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7525639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7525777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7525956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7526279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7526593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7526761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7527132Z STAGE:2022-11-23 02:03:51 39155:39155 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7527288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7527537Z STAGE:2022-11-23 02:03:51 39154:39154 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7527824Z STAGE:2022-11-23 02:03:51 39155:39155 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7528086Z STAGE:2022-11-23 02:03:51 39155:39155 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7528339Z STAGE:2022-11-23 02:03:51 39154:39154 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7528602Z STAGE:2022-11-23 02:03:51 39154:39154 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7528662Z ok (3.740s) 2022-11-23T02:05:25.7528667Z 2022-11-23T02:05:25.7528863Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7528932Z Ran 1 test in 3.740s 2022-11-23T02:05:25.7528937Z 2022-11-23T02:05:25.7528991Z OK 2022-11-23T02:05:25.7528995Z 2022-11-23T02:05:25.7529072Z Generating XML reports... 2022-11-23T02:05:25.7529417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020349.xml 2022-11-23T02:05:25.7529653Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7529929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7530084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7530371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7530499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7530505Z 2022-11-23T02:05:25.7530571Z Running tests... 2022-11-23T02:05:25.7530767Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7530935Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s) 2022-11-23T02:05:25.7530941Z 2022-11-23T02:05:25.7531137Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7531206Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7531211Z 2022-11-23T02:05:25.7531275Z OK (skipped=1) 2022-11-23T02:05:25.7531280Z 2022-11-23T02:05:25.7531351Z Generating XML reports... 2022-11-23T02:05:25.7531691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020356.xml 2022-11-23T02:05:25.7531924Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7532201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7532319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7532603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7532732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7532739Z 2022-11-23T02:05:25.7532803Z Running tests... 2022-11-23T02:05:25.7532996Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7533187Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s) 2022-11-23T02:05:25.7533192Z 2022-11-23T02:05:25.7533386Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7533455Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7533459Z 2022-11-23T02:05:25.7533522Z OK (skipped=1) 2022-11-23T02:05:25.7533527Z 2022-11-23T02:05:25.7533603Z Generating XML reports... 2022-11-23T02:05:25.7533945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020359.xml 2022-11-23T02:05:25.7534180Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7534458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7534608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7534893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7535025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7535029Z 2022-11-23T02:05:25.7535095Z Running tests... 2022-11-23T02:05:25.7535294Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7535476Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-11-23T02:05:25.7535481Z 2022-11-23T02:05:25.7535666Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7535733Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7535738Z 2022-11-23T02:05:25.7535802Z OK (skipped=1) 2022-11-23T02:05:25.7535807Z 2022-11-23T02:05:25.7535888Z Generating XML reports... 2022-11-23T02:05:25.7536228Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020401.xml 2022-11-23T02:05:25.7536459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7536771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7536889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7537173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7537302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7537307Z 2022-11-23T02:05:25.7537372Z Running tests... 2022-11-23T02:05:25.7537566Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7537806Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39517 2022-11-23T02:05:25.7537964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39518 2022-11-23T02:05:25.7538159Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7538436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7538553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7538838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7538966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7539138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7539415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7539530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7539815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7539940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7540110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7540416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7540717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7540875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7541166Z STAGE:2022-11-23 02:04:07 39518:39518 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7541324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7541576Z STAGE:2022-11-23 02:04:07 39517:39517 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7541832Z STAGE:2022-11-23 02:04:07 39518:39518 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7542095Z STAGE:2022-11-23 02:04:07 39518:39518 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7542347Z STAGE:2022-11-23 02:04:07 39517:39517 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7542610Z STAGE:2022-11-23 02:04:07 39517:39517 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7542670Z ok (3.852s) 2022-11-23T02:05:25.7542675Z 2022-11-23T02:05:25.7542876Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7542944Z Ran 1 test in 3.853s 2022-11-23T02:05:25.7542949Z 2022-11-23T02:05:25.7543002Z OK 2022-11-23T02:05:25.7543006Z 2022-11-23T02:05:25.7543082Z Generating XML reports... 2022-11-23T02:05:25.7543454Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020404.xml 2022-11-23T02:05:25.7543689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7543969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7544089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7544373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7544498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7544511Z 2022-11-23T02:05:25.7544572Z Running tests... 2022-11-23T02:05:25.7544770Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7544999Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39706 2022-11-23T02:05:25.7545153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39707 2022-11-23T02:05:25.7545343Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7545620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7545736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7546020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7546151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7546322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7546600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7546720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7547003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7547134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7547308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7547610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7547910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7548102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7548260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7548324Z ok (3.939s) 2022-11-23T02:05:25.7548329Z 2022-11-23T02:05:25.7548528Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7548591Z Ran 1 test in 3.940s 2022-11-23T02:05:25.7548603Z 2022-11-23T02:05:25.7548651Z OK 2022-11-23T02:05:25.7548655Z 2022-11-23T02:05:25.7548731Z Generating XML reports... 2022-11-23T02:05:25.7549074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020411.xml 2022-11-23T02:05:25.7549307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7549587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7549705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7549988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7550148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7550153Z 2022-11-23T02:05:25.7550220Z Running tests... 2022-11-23T02:05:25.7550418Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7550665Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39889 2022-11-23T02:05:25.7550817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39890 2022-11-23T02:05:25.7551008Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7551285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7551404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7551690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7551819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7551991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7552263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7552380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7552665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7552794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7552963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7553266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7553562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7553721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7553976Z STAGE:2022-11-23 02:04:21 39890:39890 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7554136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7554388Z STAGE:2022-11-23 02:04:21 39889:39889 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7554675Z STAGE:2022-11-23 02:04:21 39889:39889 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7554936Z STAGE:2022-11-23 02:04:21 39889:39889 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7555191Z STAGE:2022-11-23 02:04:21 39890:39890 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7555451Z STAGE:2022-11-23 02:04:21 39890:39890 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7555511Z ok (3.855s) 2022-11-23T02:05:25.7555516Z 2022-11-23T02:05:25.7555712Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7555783Z Ran 1 test in 3.855s 2022-11-23T02:05:25.7555788Z 2022-11-23T02:05:25.7555843Z OK 2022-11-23T02:05:25.7555848Z 2022-11-23T02:05:25.7555924Z Generating XML reports... 2022-11-23T02:05:25.7556317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020418.xml 2022-11-23T02:05:25.7556555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7556834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7556990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7557280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7557407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7557412Z 2022-11-23T02:05:25.7557477Z Running tests... 2022-11-23T02:05:25.7557668Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7557912Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40078 2022-11-23T02:05:25.7558068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40079 2022-11-23T02:05:25.7558256Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7558531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7558649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7558933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7559061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7559232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7559510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7559630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7559914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7560044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7560218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7560521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7560818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7560976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7561232Z STAGE:2022-11-23 02:04:28 40079:40079 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7561448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7561702Z STAGE:2022-11-23 02:04:28 40078:40078 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:05:25.7561957Z STAGE:2022-11-23 02:04:28 40078:40078 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7562222Z STAGE:2022-11-23 02:04:28 40078:40078 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7562475Z STAGE:2022-11-23 02:04:28 40079:40079 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:05:25.7562731Z STAGE:2022-11-23 02:04:28 40079:40079 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:05:25.7562791Z ok (3.939s) 2022-11-23T02:05:25.7562796Z 2022-11-23T02:05:25.7562991Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7563059Z Ran 1 test in 3.939s 2022-11-23T02:05:25.7563066Z 2022-11-23T02:05:25.7563119Z OK 2022-11-23T02:05:25.7563124Z 2022-11-23T02:05:25.7563199Z Generating XML reports... 2022-11-23T02:05:25.7563540Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020425.xml 2022-11-23T02:05:25.7563806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7564089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7564205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7564488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7564616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7564622Z 2022-11-23T02:05:25.7564688Z Running tests... 2022-11-23T02:05:25.7564889Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7565123Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40267 2022-11-23T02:05:25.7565274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40268 2022-11-23T02:05:25.7565467Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7565742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7565860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7566144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7566273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7566447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7566719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7566838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7567122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7567249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7567417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7567719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7568021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7568215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7568371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7568432Z ok (3.949s) 2022-11-23T02:05:25.7568437Z 2022-11-23T02:05:25.7568635Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7568705Z Ran 1 test in 3.950s 2022-11-23T02:05:25.7568710Z 2022-11-23T02:05:25.7568762Z OK 2022-11-23T02:05:25.7568767Z 2022-11-23T02:05:25.7568845Z Generating XML reports... 2022-11-23T02:05:25.7569187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020432.xml 2022-11-23T02:05:25.7569420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7569697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7569819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7570104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7570236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7570290Z 2022-11-23T02:05:25.7570358Z Running tests... 2022-11-23T02:05:25.7570557Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7570790Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40612 2022-11-23T02:05:25.7570943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40613 2022-11-23T02:05:25.7571132Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7571409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7571528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7571811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7571943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7572114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7572390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7572508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7572792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7572924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7573098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7573399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7573697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7573857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7574013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7574073Z ok (4.236s) 2022-11-23T02:05:25.7574078Z 2022-11-23T02:05:25.7574274Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7574344Z Ran 1 test in 4.237s 2022-11-23T02:05:25.7574349Z 2022-11-23T02:05:25.7574401Z OK 2022-11-23T02:05:25.7574405Z 2022-11-23T02:05:25.7574519Z Generating XML reports... 2022-11-23T02:05:25.7574863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020439.xml 2022-11-23T02:05:25.7575091Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7575368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7575485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7575769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7575899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7575903Z 2022-11-23T02:05:25.7575968Z Running tests... 2022-11-23T02:05:25.7576161Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7576393Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40959 2022-11-23T02:05:25.7576547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40960 2022-11-23T02:05:25.7576769Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7577047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7577165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7577450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7577579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7577751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7578027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7578147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7578433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7578560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7578731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7579034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7579338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7579499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7579655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7579716Z ok (6.246s) 2022-11-23T02:05:25.7579721Z 2022-11-23T02:05:25.7579918Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7579988Z Ran 1 test in 6.246s 2022-11-23T02:05:25.7579995Z 2022-11-23T02:05:25.7580048Z OK 2022-11-23T02:05:25.7580053Z 2022-11-23T02:05:25.7580130Z Generating XML reports... 2022-11-23T02:05:25.7580475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020446.xml 2022-11-23T02:05:25.7580708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7580983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7581100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7581420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7581547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7581552Z 2022-11-23T02:05:25.7581617Z Running tests... 2022-11-23T02:05:25.7581816Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7582048Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41152 2022-11-23T02:05:25.7582200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41153 2022-11-23T02:05:25.7582388Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7582663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7582781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7583066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7583196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7583398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7583671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7583788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7584071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7584198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7584369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7584673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7584972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7585131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7585310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpch6qnt3w 2022-11-23T02:05:25.7585501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpch6qnt3w/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7585658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7585834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82354gkl 2022-11-23T02:05:25.7586021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82354gkl/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7586084Z ok (3.939s) 2022-11-23T02:05:25.7586089Z 2022-11-23T02:05:25.7586288Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7586358Z Ran 1 test in 3.939s 2022-11-23T02:05:25.7586362Z 2022-11-23T02:05:25.7586416Z OK 2022-11-23T02:05:25.7586421Z 2022-11-23T02:05:25.7586497Z Generating XML reports... 2022-11-23T02:05:25.7586839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020456.xml 2022-11-23T02:05:25.7587073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7587350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7587466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7587746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7587911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7587916Z 2022-11-23T02:05:25.7587980Z Running tests... 2022-11-23T02:05:25.7588176Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7588402Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41339 2022-11-23T02:05:25.7588552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41340 2022-11-23T02:05:25.7588741Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7589015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7589134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7589419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7589545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7589716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7590026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7590143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7590430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7590558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7590730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7591034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7591333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7591497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7591655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7591837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfoqnrkoi 2022-11-23T02:05:25.7592027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfoqnrkoi/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7592199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8u03ysuy 2022-11-23T02:05:25.7592386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8u03ysuy/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7592448Z ok (4.150s) 2022-11-23T02:05:25.7592453Z 2022-11-23T02:05:25.7592650Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7592719Z Ran 1 test in 4.151s 2022-11-23T02:05:25.7592724Z 2022-11-23T02:05:25.7592777Z OK 2022-11-23T02:05:25.7592781Z 2022-11-23T02:05:25.7592857Z Generating XML reports... 2022-11-23T02:05:25.7593203Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020503.xml 2022-11-23T02:05:25.7593436Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7593715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7593833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7594114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7594277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7594282Z 2022-11-23T02:05:25.7594347Z Running tests... 2022-11-23T02:05:25.7594542Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7594801Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41522 2022-11-23T02:05:25.7594953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41523 2022-11-23T02:05:25.7595142Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:05:25.7595418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7595535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7595821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7595952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7596167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:05:25.7596694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7596989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7597112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7597411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7597545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7597722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:05:25.7598042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:05:25.7598212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:05:25.7598383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:05:25.7598574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip4yq60u 2022-11-23T02:05:25.7598778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip4yq60u/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7598964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpty6zxfsw 2022-11-23T02:05:25.7599162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpty6zxfsw/_remote_module_non_scriptable.py 2022-11-23T02:05:25.7599787Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:05:25.7600401Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:05:25.7600506Z ok (6.380s) 2022-11-23T02:05:25.7600511Z 2022-11-23T02:05:25.7600725Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7600797Z Ran 1 test in 6.380s 2022-11-23T02:05:25.7600803Z 2022-11-23T02:05:25.7600859Z OK 2022-11-23T02:05:25.7600863Z 2022-11-23T02:05:25.7600949Z Generating XML reports... 2022-11-23T02:05:25.7601312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020510.xml 2022-11-23T02:05:25.7601558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7601852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7601976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7602278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7602416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7602421Z 2022-11-23T02:05:25.7602489Z Running tests... 2022-11-23T02:05:25.7602695Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7603185Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:05:25.7603191Z 2022-11-23T02:05:25.7603399Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7603472Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7603477Z 2022-11-23T02:05:25.7603539Z OK (skipped=1) 2022-11-23T02:05:25.7603548Z 2022-11-23T02:05:25.7603625Z Generating XML reports... 2022-11-23T02:05:25.7603984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020519.xml 2022-11-23T02:05:25.7604232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:05:25.7604521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:05:25.7604645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:05:25.7604944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:05:25.7605081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:05:25.7605086Z 2022-11-23T02:05:25.7605158Z Running tests... 2022-11-23T02:05:25.7605363Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7605737Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.001s) 2022-11-23T02:05:25.7605743Z 2022-11-23T02:05:25.7605950Z ---------------------------------------------------------------------- 2022-11-23T02:05:25.7606026Z Ran 1 test in 0.002s 2022-11-23T02:05:25.7606031Z 2022-11-23T02:05:25.7606097Z OK (skipped=1) 2022-11-23T02:05:25.7606101Z 2022-11-23T02:05:25.7606183Z Generating XML reports... 2022-11-23T02:05:25.7606546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020522.xml 2022-11-23T02:05:25.7606552Z 2022-11-23T02:05:25.7606905Z ##[endgroup] 2022-11-23T02:05:25.7607369Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_ckaa7sv5) 2022-11-23T02:05:25.7607374Z 2022-11-23T02:05:25.7607521Z Running distributed tests for the gloo backend with file init_method in shard 1 of 2 2022-11-23T02:05:25.7607961Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:05:25.544604] 2022-11-23T02:32:19.4727466Z 2022-11-23T02:32:19.4728745Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T02:32:19.4731024Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_jdx3nt9e) 2022-11-23T02:32:19.4756971Z 2022-11-23T02:32:19.4889946Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-11-23T02:32:19.4962852Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4963633Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4969681Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4970335Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4970971Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4971702Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4972642Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4973364Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4974144Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4974978Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4975786Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4976615Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4977397Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4978124Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4978825Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4979601Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4980324Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4980973Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4981750Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4982438Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4983194Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4983947Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4984540Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4985112Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4985759Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4986422Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4987055Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4987750Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4988358Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4988940Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4989489Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4990100Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4990663Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4991248Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4991892Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4992534Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4993185Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4993801Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4994398Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4995015Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4995631Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4996377Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4997073Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4997724Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4998382Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4999137Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.4999818Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5000488Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5001100Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5001911Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5002358Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5002790Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5003214Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5003657Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5004104Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5004540Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5004982Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5005396Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5005818Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5006322Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5006748Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5007179Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5007585Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5007967Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5008373Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5008799Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5009218Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5009625Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5010019Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5010398Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5010804Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5011213Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5011621Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5012048Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5012447Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5012830Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5013207Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5013605Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5014022Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5014435Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5014845Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5015247Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5015651Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5016096Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5016548Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5017009Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5017479Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5017950Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5018469Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5018915Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5019364Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5019815Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5020273Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5020738Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5021213Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5021689Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5022144Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5022611Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5023050Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5023452Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5023841Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5024322Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5024688Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5025080Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5025489Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5025888Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5026283Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5026695Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5027102Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5027518Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5027927Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5028348Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5028796Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5029228Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5029656Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5030072Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5030496Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5030941Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5031381Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5031813Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5032212Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5032585Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5032986Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5033387Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5033791Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5034200Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5034672Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5035207Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5035658Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5036165Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5036666Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5037114Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5037582Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5038072Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5038528Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5038959Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5039419Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5039845Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5040233Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5040639Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5041070Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5041481Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5041933Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5042441Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5042876Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5043304Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5043781Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5044298Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5044874Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5045526Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5046170Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5046807Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5047446Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5048079Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5048713Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5049341Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5049908Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5050412Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5050872Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5051269Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5051687Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5052112Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5052525Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5052964Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5053457Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5053932Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5054419Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5054849Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5055257Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5055683Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5056125Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5056571Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5056999Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5057424Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5057874Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5058325Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5058761Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5059211Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5059637Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5060074Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5060489Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5060927Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5061406Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5061858Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5062261Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5062654Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5063067Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5063504Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5063940Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5064332Z test_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5064693Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5065076Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5065466Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5065857Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5066241Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5066641Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5067029Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5067402Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5067772Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5068162Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5068575Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5068978Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5069346Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5069705Z test_isend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5070093Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5070511Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5070936Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5071405Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5071918Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5072333Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5072768Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5073223Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5073662Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5074102Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5074533Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5074948Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5075370Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5075795Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5076298Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5076717Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5077206Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5077861Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5078345Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5078810Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5079289Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5079758Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5080205Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5080642Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5081088Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5081534Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5082005Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5082507Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5083047Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5083553Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5083991Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5084387Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5084804Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5085225Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5085629Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5086022Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5086429Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5086828Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5087192Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5087562Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5087946Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5088336Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5088748Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5089165Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5089596Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5089976Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5090374Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5090773Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5091149Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5091528Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5091903Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5092287Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5092685Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5093092Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5093487Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5093885Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5094258Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5094639Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5095076Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5095539Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5096026Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5096434Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5096847Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5097276Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5097703Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5098112Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5098542Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5098996Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5099421Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5099823Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5100249Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5100660Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5101053Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5101487Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5101956Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5102413Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5103178Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5103868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5104342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5104967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5105463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5105697Z 2022-11-23T02:32:19.5105797Z Running tests... 2022-11-23T02:32:19.5106239Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5106876Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41889 2022-11-23T02:32:19.5107520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41890 2022-11-23T02:32:19.5108111Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5108820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5109281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5109906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5110392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5110866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5111527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5111983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5112598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5113082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5113555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5114317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5115073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5115609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5116172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5116722Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:32:19.5117649Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:32:19.5118357Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:32:19.5119257Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:32:19.5119954Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:32:19.5120849Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:32:19.5121551Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:32:19.5122448Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:32:19.5123127Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:32:19.5124018Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:32:19.5124707Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:32:19.5125602Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:32:19.5126297Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:32:19.5126949Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:32:19.5127847Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:32:19.5128830Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:32:19.5129335Z ok (4.333s) 2022-11-23T02:32:19.5129480Z 2022-11-23T02:32:19.5129757Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5130095Z Ran 1 test in 4.334s 2022-11-23T02:32:19.5130248Z 2022-11-23T02:32:19.5130331Z OK 2022-11-23T02:32:19.5130455Z 2022-11-23T02:32:19.5130570Z Generating XML reports... 2022-11-23T02:32:19.5131231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020528.xml 2022-11-23T02:32:19.5131946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5132614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5133120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5133746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5134234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5134466Z 2022-11-23T02:32:19.5134566Z Running tests... 2022-11-23T02:32:19.5135001Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5135528Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s) 2022-11-23T02:32:19.5135852Z 2022-11-23T02:32:19.5136137Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5136456Z Ran 1 test in 0.003s 2022-11-23T02:32:19.5136612Z 2022-11-23T02:32:19.5136709Z OK (skipped=1) 2022-11-23T02:32:19.5136857Z 2022-11-23T02:32:19.5136974Z Generating XML reports... 2022-11-23T02:32:19.5137630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020536.xml 2022-11-23T02:32:19.5138330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5138996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5139459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5140068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5140560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5140793Z 2022-11-23T02:32:19.5140890Z Running tests... 2022-11-23T02:32:19.5141318Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5141889Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42134 2022-11-23T02:32:19.5142460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42135 2022-11-23T02:32:19.5142982Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5143684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5144137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5144751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5145317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5145791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5146513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5147230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5147694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5148298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5148781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5149251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5149962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5150511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5151043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5151396Z ok (3.750s) 2022-11-23T02:32:19.5151540Z 2022-11-23T02:32:19.5151833Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5152151Z Ran 1 test in 3.750s 2022-11-23T02:32:19.5152303Z 2022-11-23T02:32:19.5152386Z OK 2022-11-23T02:32:19.5152514Z 2022-11-23T02:32:19.5152631Z Generating XML reports... 2022-11-23T02:32:19.5153287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020539.xml 2022-11-23T02:32:19.5153997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5154665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5155126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5155736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5156280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5156506Z 2022-11-23T02:32:19.5156623Z Running tests... 2022-11-23T02:32:19.5157050Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5158306Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.506s) 2022-11-23T02:32:19.5158969Z 2022-11-23T02:32:19.5159246Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5159566Z Ran 1 test in 0.506s 2022-11-23T02:32:19.5159713Z 2022-11-23T02:32:19.5159802Z OK (skipped=1) 2022-11-23T02:32:19.5159944Z 2022-11-23T02:32:19.5160058Z Generating XML reports... 2022-11-23T02:32:19.5160699Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020546.xml 2022-11-23T02:32:19.5161388Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5162038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5162491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5163097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5163639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5163867Z 2022-11-23T02:32:19.5163962Z Running tests... 2022-11-23T02:32:19.5164382Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5164976Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42375 2022-11-23T02:32:19.5165570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42376 2022-11-23T02:32:19.5166085Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5166769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5167217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5167819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5168290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5168742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5169490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5170194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5170639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5171246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5171721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5172187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5172881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5173407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5173918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpfbu8eh 2022-11-23T02:32:19.5174472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpfbu8eh/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5174984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5175497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsct5fbn 2022-11-23T02:32:19.5176047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsct5fbn/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5176576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5177075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5177413Z ok (4.366s) 2022-11-23T02:32:19.5177546Z 2022-11-23T02:32:19.5177839Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5178159Z Ran 1 test in 4.366s 2022-11-23T02:32:19.5178308Z 2022-11-23T02:32:19.5178382Z OK 2022-11-23T02:32:19.5178499Z 2022-11-23T02:32:19.5178606Z Generating XML reports... 2022-11-23T02:32:19.5179249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020549.xml 2022-11-23T02:32:19.5179937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5180594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5181114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5181724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5182196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5182418Z 2022-11-23T02:32:19.5182519Z Running tests... 2022-11-23T02:32:19.5182935Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5183535Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42562 2022-11-23T02:32:19.5184152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42563 2022-11-23T02:32:19.5184677Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5185372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5185821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5186419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5186937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5187395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5188060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5188500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5189098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5189562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5190019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5190718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5191449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5191976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5192486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1f2fqopc 2022-11-23T02:32:19.5193036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1f2fqopc/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5193551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5194057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7geuwx3 2022-11-23T02:32:19.5194600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7geuwx3/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5195139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5195633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5195978Z ok (3.839s) 2022-11-23T02:32:19.5196166Z 2022-11-23T02:32:19.5196453Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5196781Z Ran 1 test in 3.839s 2022-11-23T02:32:19.5196929Z 2022-11-23T02:32:19.5197010Z OK 2022-11-23T02:32:19.5197134Z 2022-11-23T02:32:19.5197249Z Generating XML reports... 2022-11-23T02:32:19.5197902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020557.xml 2022-11-23T02:32:19.5198596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5199304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5199757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5200360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5200834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5201055Z 2022-11-23T02:32:19.5201147Z Running tests... 2022-11-23T02:32:19.5201559Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5202158Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42749 2022-11-23T02:32:19.5202753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42750 2022-11-23T02:32:19.5203263Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5203953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5204405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5205053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5205527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5205987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5206649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5207100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5207710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5208186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5208645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5209365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5210117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5210665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5211130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5211638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbyy1e10d 2022-11-23T02:32:19.5212186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbyy1e10d/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5212732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8civpkl 2022-11-23T02:32:19.5213278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8civpkl/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5213801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5214289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5214775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5215253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5215729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5216201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5216732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5217256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5217819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5218377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5218884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5219437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5219854Z ok (6.946s) 2022-11-23T02:32:19.5220032Z 2022-11-23T02:32:19.5220368Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5220768Z Ran 1 test in 6.947s 2022-11-23T02:32:19.5220956Z 2022-11-23T02:32:19.5221075Z OK 2022-11-23T02:32:19.5221238Z 2022-11-23T02:32:19.5221384Z Generating XML reports... 2022-11-23T02:32:19.5222074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020603.xml 2022-11-23T02:32:19.5222853Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5223637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5224172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5224881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5225460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5225724Z 2022-11-23T02:32:19.5225855Z Running tests... 2022-11-23T02:32:19.5226370Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5227054Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42942 2022-11-23T02:32:19.5227789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42943 2022-11-23T02:32:19.5228404Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5229181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5229739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5230444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5231015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5231567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5232344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5232888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5233533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5234085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5234619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5235401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5236276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5236892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5237528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5238122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pd_4uh0 2022-11-23T02:32:19.5238707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pd_4uh0/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5239324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gjo8_5v 2022-11-23T02:32:19.5239947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gjo8_5v/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5240583Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5241223Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5241814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5242383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5242903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5243527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5243949Z ok (4.452s) 2022-11-23T02:32:19.5244125Z 2022-11-23T02:32:19.5244463Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5244860Z Ran 1 test in 4.452s 2022-11-23T02:32:19.5245051Z 2022-11-23T02:32:19.5245165Z OK 2022-11-23T02:32:19.5245322Z 2022-11-23T02:32:19.5245467Z Generating XML reports... 2022-11-23T02:32:19.5246194Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020614.xml 2022-11-23T02:32:19.5246963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5247696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5248228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5248923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5249484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5249745Z 2022-11-23T02:32:19.5249875Z Running tests... 2022-11-23T02:32:19.5250333Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5251028Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43133 2022-11-23T02:32:19.5251722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43134 2022-11-23T02:32:19.5252319Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5253087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5253650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5254375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5254943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5255441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5256366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5256868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5257635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5258196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5258778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5259589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5260484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5261107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5261699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5262259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vjfzumz 2022-11-23T02:32:19.5262905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vjfzumz/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5263539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrmodwnq 2022-11-23T02:32:19.5264227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrmodwnq/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5279505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5280026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5280527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5281018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5281504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5281982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5282467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5282950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5283293Z ok (4.641s) 2022-11-23T02:32:19.5283434Z 2022-11-23T02:32:19.5283795Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5284124Z Ran 1 test in 4.641s 2022-11-23T02:32:19.5284277Z 2022-11-23T02:32:19.5284348Z OK 2022-11-23T02:32:19.5284467Z 2022-11-23T02:32:19.5284578Z Generating XML reports... 2022-11-23T02:32:19.5285242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020621.xml 2022-11-23T02:32:19.5285943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5286610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5287070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5287683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5288162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5288387Z 2022-11-23T02:32:19.5288479Z Running tests... 2022-11-23T02:32:19.5288909Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5289551Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43324 2022-11-23T02:32:19.5290182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43325 2022-11-23T02:32:19.5290702Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5291552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5292014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5292625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5293104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5293569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5294272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5294975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5295431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5296046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5296515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5297034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5297735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5298274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5298757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5299270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk64qklze 2022-11-23T02:32:19.5299823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk64qklze/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5300375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcoueyfys 2022-11-23T02:32:19.5300915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcoueyfys/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5301444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5301936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5302281Z ok (4.741s) 2022-11-23T02:32:19.5302419Z 2022-11-23T02:32:19.5302709Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5303042Z Ran 1 test in 4.742s 2022-11-23T02:32:19.5303193Z 2022-11-23T02:32:19.5303275Z OK 2022-11-23T02:32:19.5303387Z 2022-11-23T02:32:19.5303500Z Generating XML reports... 2022-11-23T02:32:19.5304158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020629.xml 2022-11-23T02:32:19.5304860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5305518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5305978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5306591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5307073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5307299Z 2022-11-23T02:32:19.5307385Z Running tests... 2022-11-23T02:32:19.5307809Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5308451Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43515 2022-11-23T02:32:19.5309177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43516 2022-11-23T02:32:19.5309693Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5310396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5310859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5311477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5311949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5312422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5313104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5313568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5314188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5314678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5315208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5315921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5316743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5317296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5317795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5318324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ah_f3s5 2022-11-23T02:32:19.5318897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ah_f3s5/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5319460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh7gskpk 2022-11-23T02:32:19.5320021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh7gskpk/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5320545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5321114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5321608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5322096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5322451Z ok (6.346s) 2022-11-23T02:32:19.5322593Z 2022-11-23T02:32:19.5322887Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5323221Z Ran 1 test in 6.347s 2022-11-23T02:32:19.5323365Z 2022-11-23T02:32:19.5323447Z OK 2022-11-23T02:32:19.5323573Z 2022-11-23T02:32:19.5323685Z Generating XML reports... 2022-11-23T02:32:19.5324343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020637.xml 2022-11-23T02:32:19.5325049Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5325709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5326171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5326783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5327250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5327537Z 2022-11-23T02:32:19.5327632Z Running tests... 2022-11-23T02:32:19.5328063Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5328690Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43708 2022-11-23T02:32:19.5329314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43709 2022-11-23T02:32:19.5329839Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5330537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5330983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5331592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5332083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5332553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5333281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5333738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5334353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5334833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5335285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5335998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5336745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5337286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5337775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5338295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkk0j3cnc 2022-11-23T02:32:19.5338848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkk0j3cnc/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5339386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxd1fbsaj 2022-11-23T02:32:19.5339942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxd1fbsaj/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5340476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5340976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5341469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5341959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5342313Z ok (6.646s) 2022-11-23T02:32:19.5342455Z 2022-11-23T02:32:19.5342737Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5343065Z Ran 1 test in 6.647s 2022-11-23T02:32:19.5343219Z 2022-11-23T02:32:19.5343301Z OK 2022-11-23T02:32:19.5343425Z 2022-11-23T02:32:19.5343538Z Generating XML reports... 2022-11-23T02:32:19.5344196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020646.xml 2022-11-23T02:32:19.5344901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5345617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5346069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5346685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5347178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5347405Z 2022-11-23T02:32:19.5347501Z Running tests... 2022-11-23T02:32:19.5347926Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5348731Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43901 2022-11-23T02:32:19.5349387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43902 2022-11-23T02:32:19.5349907Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5350599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5351061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5351720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5352202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5352668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5353376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5354091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5354555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5355157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5355638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5356173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5356883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5357424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5357912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5358431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiac5gt0f 2022-11-23T02:32:19.5358983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiac5gt0f/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5359541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplak7c199 2022-11-23T02:32:19.5360095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplak7c199/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5360665Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5361228Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5361746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5362242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5362739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5363279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5363626Z ok (4.356s) 2022-11-23T02:32:19.5363768Z 2022-11-23T02:32:19.5364063Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5364396Z Ran 1 test in 4.356s 2022-11-23T02:32:19.5364551Z 2022-11-23T02:32:19.5364631Z OK 2022-11-23T02:32:19.5364758Z 2022-11-23T02:32:19.5364868Z Generating XML reports... 2022-11-23T02:32:19.5365523Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020656.xml 2022-11-23T02:32:19.5366228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5366888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5367350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5367969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5368456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5368684Z 2022-11-23T02:32:19.5368780Z Running tests... 2022-11-23T02:32:19.5369198Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5370554Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:32:19.5371253Z 2022-11-23T02:32:19.5371527Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5371859Z Ran 1 test in 0.510s 2022-11-23T02:32:19.5372017Z 2022-11-23T02:32:19.5372118Z OK (skipped=1) 2022-11-23T02:32:19.5372265Z 2022-11-23T02:32:19.5372379Z Generating XML reports... 2022-11-23T02:32:19.5373034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020703.xml 2022-11-23T02:32:19.5373741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5374400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5374851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5375466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5375953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5376180Z 2022-11-23T02:32:19.5376277Z Running tests... 2022-11-23T02:32:19.5376706Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5377312Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44150 2022-11-23T02:32:19.5377920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44151 2022-11-23T02:32:19.5378450Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5379131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5379595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5380216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5380701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5381173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5381898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5382361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5382967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5383449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5383918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5384627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5385367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5385913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5386397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5386772Z ok (3.838s) 2022-11-23T02:32:19.5386904Z 2022-11-23T02:32:19.5387238Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5387566Z Ran 1 test in 3.839s 2022-11-23T02:32:19.5387721Z 2022-11-23T02:32:19.5387801Z OK 2022-11-23T02:32:19.5387926Z 2022-11-23T02:32:19.5388041Z Generating XML reports... 2022-11-23T02:32:19.5388704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020707.xml 2022-11-23T02:32:19.5389415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5390077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5390528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5391141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5391626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5391856Z 2022-11-23T02:32:19.5391957Z Running tests... 2022-11-23T02:32:19.5392391Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5393698Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:32:19.5394382Z 2022-11-23T02:32:19.5394665Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5394999Z Ran 1 test in 0.509s 2022-11-23T02:32:19.5395155Z 2022-11-23T02:32:19.5395238Z OK (skipped=1) 2022-11-23T02:32:19.5395388Z 2022-11-23T02:32:19.5395503Z Generating XML reports... 2022-11-23T02:32:19.5396229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020714.xml 2022-11-23T02:32:19.5396940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5397602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5398065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5398678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5399169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5399446Z 2022-11-23T02:32:19.5399544Z Running tests... 2022-11-23T02:32:19.5399977Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5400556Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44391 2022-11-23T02:32:19.5401143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44392 2022-11-23T02:32:19.5401659Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5402350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5402812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5403412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5403895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5404361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5405023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5405535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5406148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5406627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5407087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5407804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5408533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5409075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5409557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5409909Z ok (4.352s) 2022-11-23T02:32:19.5410055Z 2022-11-23T02:32:19.5410342Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5410676Z Ran 1 test in 4.352s 2022-11-23T02:32:19.5410820Z 2022-11-23T02:32:19.5410900Z OK 2022-11-23T02:32:19.5411022Z 2022-11-23T02:32:19.5411138Z Generating XML reports... 2022-11-23T02:32:19.5411798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020717.xml 2022-11-23T02:32:19.5412499Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5413165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5413630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5414242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5414712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5414946Z 2022-11-23T02:32:19.5415046Z Running tests... 2022-11-23T02:32:19.5415477Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5415974Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-11-23T02:32:19.5416266Z 2022-11-23T02:32:19.5416555Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5416881Z Ran 1 test in 0.002s 2022-11-23T02:32:19.5417036Z 2022-11-23T02:32:19.5417135Z OK (skipped=1) 2022-11-23T02:32:19.5417322Z 2022-11-23T02:32:19.5417434Z Generating XML reports... 2022-11-23T02:32:19.5418084Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020725.xml 2022-11-23T02:32:19.5418785Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5419445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5419905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5420523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5421008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5421237Z 2022-11-23T02:32:19.5421324Z Running tests... 2022-11-23T02:32:19.5421753Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5422197Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5422757Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44632 2022-11-23T02:32:19.5423305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44633 2022-11-23T02:32:19.5423879Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5424576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5425033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5425643Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5426125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5426605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5427317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5428025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5428487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5429105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5429574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5430044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5430758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5431301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5431814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdh_0h321 2022-11-23T02:32:19.5432379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdh_0h321/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5432903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5433417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz1j0tmd 2022-11-23T02:32:19.5433958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz1j0tmd/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5434347Z ok (3.840s) 2022-11-23T02:32:19.5434487Z 2022-11-23T02:32:19.5434776Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5435103Z Ran 1 test in 3.841s 2022-11-23T02:32:19.5435265Z 2022-11-23T02:32:19.5435398Z OK 2022-11-23T02:32:19.5435521Z 2022-11-23T02:32:19.5435635Z Generating XML reports... 2022-11-23T02:32:19.5436338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020728.xml 2022-11-23T02:32:19.5437045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5437707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5438169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5438783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5439265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5439492Z 2022-11-23T02:32:19.5439589Z Running tests... 2022-11-23T02:32:19.5440023Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5440478Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5441061Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44819 2022-11-23T02:32:19.5441726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44820 2022-11-23T02:32:19.5442251Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5442947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5443405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5444020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5444495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5444970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5445676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5446387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5446852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5447461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5447942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5448409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5449105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5449649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5450165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoyfx6q30 2022-11-23T02:32:19.5450730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoyfx6q30/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5451256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5451769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuewo1vm8 2022-11-23T02:32:19.5452479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuewo1vm8/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5453009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5453496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5453938Z ok (3.865s) 2022-11-23T02:32:19.5454076Z 2022-11-23T02:32:19.5454370Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5454702Z Ran 1 test in 3.865s 2022-11-23T02:32:19.5454856Z 2022-11-23T02:32:19.5454937Z OK 2022-11-23T02:32:19.5455061Z 2022-11-23T02:32:19.5455181Z Generating XML reports... 2022-11-23T02:32:19.5455824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020735.xml 2022-11-23T02:32:19.5456522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5457184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5457643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5458259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5458751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5458979Z 2022-11-23T02:32:19.5459075Z Running tests... 2022-11-23T02:32:19.5459487Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5460008Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5460632Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45006 2022-11-23T02:32:19.5461206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45007 2022-11-23T02:32:19.5461726Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5462427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5462885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5463502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5463977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5464450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5465168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5465878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5466335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5466941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5467423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5467887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5468583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5469122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5469642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkw4q3bku 2022-11-23T02:32:19.5470200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkw4q3bku/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5470728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5471248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphhk_hq62 2022-11-23T02:32:19.5471795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphhk_hq62/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5472369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5472856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.5473200Z ok (3.842s) 2022-11-23T02:32:19.5473344Z 2022-11-23T02:32:19.5473630Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5473952Z Ran 1 test in 3.842s 2022-11-23T02:32:19.5474105Z 2022-11-23T02:32:19.5474186Z OK 2022-11-23T02:32:19.5474306Z 2022-11-23T02:32:19.5474420Z Generating XML reports... 2022-11-23T02:32:19.5475064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020742.xml 2022-11-23T02:32:19.5475765Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5476475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5477010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5477626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5478159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5478388Z 2022-11-23T02:32:19.5478486Z Running tests... 2022-11-23T02:32:19.5478926Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5479369Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:32:19.5479941Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45193 2022-11-23T02:32:19.5480482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45194 2022-11-23T02:32:19.5480996Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5481685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5482143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5482754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5483221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5483681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5484384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5485083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5485534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5486141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5486623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5487089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5487780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5488308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5488822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9io2r6ey 2022-11-23T02:32:19.5489386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9io2r6ey/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5489956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5490466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphhn8f_ei 2022-11-23T02:32:19.5491013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphhn8f_ei/_remote_module_non_scriptable.py 2022-11-23T02:32:19.5491390Z ok (3.940s) 2022-11-23T02:32:19.5491535Z 2022-11-23T02:32:19.5491823Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5492147Z Ran 1 test in 3.941s 2022-11-23T02:32:19.5492299Z 2022-11-23T02:32:19.5492375Z OK 2022-11-23T02:32:19.5492494Z 2022-11-23T02:32:19.5492603Z Generating XML reports... 2022-11-23T02:32:19.5493248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020748.xml 2022-11-23T02:32:19.5493943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5494593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5495054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5495660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5496186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5496413Z 2022-11-23T02:32:19.5496505Z Running tests... 2022-11-23T02:32:19.5496931Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5497473Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45380 2022-11-23T02:32:19.5498018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45381 2022-11-23T02:32:19.5498530Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5499227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5499679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5500292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5500769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5501229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5501892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5502334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5502944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5503419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5503884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5504580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5505314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5505847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5506481Z STAGE:2022-11-23 02:07:58 45381:45381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5506962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5507576Z STAGE:2022-11-23 02:07:58 45380:45380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5508247Z STAGE:2022-11-23 02:07:58 45381:45381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5508879Z STAGE:2022-11-23 02:07:58 45381:45381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5509511Z STAGE:2022-11-23 02:07:58 45380:45380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5510139Z STAGE:2022-11-23 02:07:58 45380:45380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5510761Z STAGE:2022-11-23 02:07:58 45380:45380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5511363Z STAGE:2022-11-23 02:07:58 45381:45381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5511973Z STAGE:2022-11-23 02:07:58 45381:45381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5512588Z STAGE:2022-11-23 02:07:58 45380:45380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5513223Z STAGE:2022-11-23 02:07:58 45381:45381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5513876Z STAGE:2022-11-23 02:07:58 45380:45380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5514366Z ok (3.840s) 2022-11-23T02:32:19.5514507Z 2022-11-23T02:32:19.5514797Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5515126Z Ran 1 test in 3.841s 2022-11-23T02:32:19.5515278Z 2022-11-23T02:32:19.5515348Z OK 2022-11-23T02:32:19.5515471Z 2022-11-23T02:32:19.5515584Z Generating XML reports... 2022-11-23T02:32:19.5516297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020755.xml 2022-11-23T02:32:19.5516999Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5517659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5518123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5518743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5519218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5519443Z 2022-11-23T02:32:19.5519539Z Running tests... 2022-11-23T02:32:19.5519969Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5520540Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45569 2022-11-23T02:32:19.5521115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45570 2022-11-23T02:32:19.5521626Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5522325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5522787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5523395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5523868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5524334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5524992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5525448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5526048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5526579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5527035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5527735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5528460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5528987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5529605Z STAGE:2022-11-23 02:08:05 45570:45570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5530097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5530711Z STAGE:2022-11-23 02:08:05 45569:45569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5531337Z STAGE:2022-11-23 02:08:05 45570:45570 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5531948Z STAGE:2022-11-23 02:08:05 45569:45569 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5532631Z STAGE:2022-11-23 02:08:05 45570:45570 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5533283Z STAGE:2022-11-23 02:08:05 45569:45569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5533893Z STAGE:2022-11-23 02:08:05 45570:45570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5534492Z STAGE:2022-11-23 02:08:05 45569:45569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5535100Z STAGE:2022-11-23 02:08:05 45570:45570 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5535979Z STAGE:2022-11-23 02:08:05 45569:45569 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:08:05 45570:45570 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5536392Z 2022-11-23T02:32:19.5536770Z STAGE:2022-11-23 02:08:05 45569:45569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5537400Z STAGE:2022-11-23 02:08:05 45569:45569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5537996Z STAGE:2022-11-23 02:08:05 45570:45570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5538611Z STAGE:2022-11-23 02:08:05 45569:45569 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5539251Z STAGE:2022-11-23 02:08:05 45569:45569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5539884Z STAGE:2022-11-23 02:08:05 45570:45570 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5540529Z STAGE:2022-11-23 02:08:05 45570:45570 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5541649Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5542298Z warnings.warn( 2022-11-23T02:32:19.5543256Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5543889Z warnings.warn( 2022-11-23T02:32:19.5544107Z ok (3.939s) 2022-11-23T02:32:19.5544246Z 2022-11-23T02:32:19.5544536Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5544876Z Ran 1 test in 3.939s 2022-11-23T02:32:19.5545085Z 2022-11-23T02:32:19.5545165Z OK 2022-11-23T02:32:19.5545288Z 2022-11-23T02:32:19.5545404Z Generating XML reports... 2022-11-23T02:32:19.5546059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020802.xml 2022-11-23T02:32:19.5546767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5547418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5547880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5548487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5548970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5549196Z 2022-11-23T02:32:19.5549292Z Running tests... 2022-11-23T02:32:19.5549885Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5550465Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45758 2022-11-23T02:32:19.5551037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45759 2022-11-23T02:32:19.5551606Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5552316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5552783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5553404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5553880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5554347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5555021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5555470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5556160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5556636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5557103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5557820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5558569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5559116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5559587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5560085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.5560607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.5561303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5562047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5562725Z STAGE:2022-11-23 02:08:12 45758:45758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5563344Z STAGE:2022-11-23 02:08:12 45759:45759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5564033Z STAGE:2022-11-23 02:08:12 45758:45758 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5564644Z STAGE:2022-11-23 02:08:12 45759:45759 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5565286Z STAGE:2022-11-23 02:08:12 45758:45758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5565947Z STAGE:2022-11-23 02:08:12 45759:45759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5566583Z STAGE:2022-11-23 02:08:12 45758:45758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5567187Z STAGE:2022-11-23 02:08:12 45759:45759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5568066Z STAGE:2022-11-23 02:08:12 45759:45759 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:08:12 45758:45758 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5568482Z 2022-11-23T02:32:19.5568862Z STAGE:2022-11-23 02:08:12 45758:45758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5569510Z STAGE:2022-11-23 02:08:12 45759:45759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5570183Z STAGE:2022-11-23 02:08:12 45758:45758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5570786Z STAGE:2022-11-23 02:08:12 45759:45759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5571399Z STAGE:2022-11-23 02:08:12 45758:45758 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5572047Z STAGE:2022-11-23 02:08:12 45758:45758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5572685Z STAGE:2022-11-23 02:08:12 45759:45759 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5573328Z STAGE:2022-11-23 02:08:12 45759:45759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5574415Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5575068Z warnings.warn( 2022-11-23T02:32:19.5576018Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5576647Z warnings.warn( 2022-11-23T02:32:19.5576867Z ok (3.958s) 2022-11-23T02:32:19.5577006Z 2022-11-23T02:32:19.5577299Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5577628Z Ran 1 test in 3.958s 2022-11-23T02:32:19.5577784Z 2022-11-23T02:32:19.5577863Z OK 2022-11-23T02:32:19.5577988Z 2022-11-23T02:32:19.5578099Z Generating XML reports... 2022-11-23T02:32:19.5578749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020809.xml 2022-11-23T02:32:19.5579457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5580109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5580771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5581421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5582017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5582286Z 2022-11-23T02:32:19.5582453Z Running tests... 2022-11-23T02:32:19.5583045Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5583709Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45953 2022-11-23T02:32:19.5584405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45954 2022-11-23T02:32:19.5585029Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5585811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5586303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5587036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5587603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5588335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5589129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5589742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5590509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5591033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5591579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5592376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5593251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5593894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5594465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5594981Z skip: Skipped due to small world size. (3.738s) 2022-11-23T02:32:19.5595287Z 2022-11-23T02:32:19.5595635Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5596000Z Ran 1 test in 3.739s 2022-11-23T02:32:19.5596265Z 2022-11-23T02:32:19.5596404Z OK (skipped=1) 2022-11-23T02:32:19.5596621Z 2022-11-23T02:32:19.5596776Z Generating XML reports... 2022-11-23T02:32:19.5597524Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020816.xml 2022-11-23T02:32:19.5598335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5599104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5599652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5611108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5611688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5611922Z 2022-11-23T02:32:19.5612018Z Running tests... 2022-11-23T02:32:19.5612507Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5613090Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46136 2022-11-23T02:32:19.5613675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46137 2022-11-23T02:32:19.5614187Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5615039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5615499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5616117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5616596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5617061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5617739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5618201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5618800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5619281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5619745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5620514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5621258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5621789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5622420Z STAGE:2022-11-23 02:08:25 46137:46137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5622920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5623536Z STAGE:2022-11-23 02:08:26 46136:46136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5624168Z STAGE:2022-11-23 02:08:26 46136:46136 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5624814Z STAGE:2022-11-23 02:08:26 46136:46136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5625453Z STAGE:2022-11-23 02:08:26 46137:46137 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5626083Z STAGE:2022-11-23 02:08:26 46137:46137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5626702Z STAGE:2022-11-23 02:08:26 46137:46137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5627313Z STAGE:2022-11-23 02:08:26 46136:46136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5627931Z STAGE:2022-11-23 02:08:26 46136:46136 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5628557Z STAGE:2022-11-23 02:08:26 46136:46136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5629193Z STAGE:2022-11-23 02:08:26 46137:46137 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5629827Z STAGE:2022-11-23 02:08:26 46137:46137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5630457Z STAGE:2022-11-23 02:08:26 46136:46136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5631064Z STAGE:2022-11-23 02:08:26 46137:46137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5631674Z STAGE:2022-11-23 02:08:26 46136:46136 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5632307Z STAGE:2022-11-23 02:08:26 46136:46136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5632944Z STAGE:2022-11-23 02:08:26 46137:46137 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5633563Z STAGE:2022-11-23 02:08:26 46137:46137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5634740Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5635383Z warnings.warn( 2022-11-23T02:32:19.5636413Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5637047Z warnings.warn( 2022-11-23T02:32:19.5637280Z ok (4.038s) 2022-11-23T02:32:19.5637424Z 2022-11-23T02:32:19.5637713Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5638047Z Ran 1 test in 4.038s 2022-11-23T02:32:19.5638210Z 2022-11-23T02:32:19.5638279Z OK 2022-11-23T02:32:19.5638401Z 2022-11-23T02:32:19.5638511Z Generating XML reports... 2022-11-23T02:32:19.5639161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020823.xml 2022-11-23T02:32:19.5639930Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5640605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5641076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5641695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5642166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5642392Z 2022-11-23T02:32:19.5642486Z Running tests... 2022-11-23T02:32:19.5642915Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5643501Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46325 2022-11-23T02:32:19.5644072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46326 2022-11-23T02:32:19.5644596Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5645290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5645750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5646350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5646827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5647291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5647960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5648417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5649023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5649498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5649962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5650658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5651394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5651990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5652774Z STAGE:2022-11-23 02:08:32 46326:46326 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5653260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5653884Z STAGE:2022-11-23 02:08:33 46325:46325 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5654517Z STAGE:2022-11-23 02:08:33 46325:46325 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5655159Z STAGE:2022-11-23 02:08:33 46325:46325 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5655787Z STAGE:2022-11-23 02:08:33 46326:46326 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5656417Z STAGE:2022-11-23 02:08:33 46326:46326 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5657504Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5658157Z warnings.warn( 2022-11-23T02:32:19.5659144Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5659782Z warnings.warn( 2022-11-23T02:32:19.5660011Z ok (3.842s) 2022-11-23T02:32:19.5660151Z 2022-11-23T02:32:19.5660441Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5660771Z Ran 1 test in 3.842s 2022-11-23T02:32:19.5660916Z 2022-11-23T02:32:19.5660993Z OK 2022-11-23T02:32:19.5661119Z 2022-11-23T02:32:19.5661230Z Generating XML reports... 2022-11-23T02:32:19.5661880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020830.xml 2022-11-23T02:32:19.5662587Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5663254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5663712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5664328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5664801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5665026Z 2022-11-23T02:32:19.5665122Z Running tests... 2022-11-23T02:32:19.5665548Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5666111Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46514 2022-11-23T02:32:19.5666673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46515 2022-11-23T02:32:19.5667194Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5667887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5668335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5668942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5669417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5669881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5670638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5671334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5671795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5672410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5672882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5673341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5674041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5674578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5675206Z STAGE:2022-11-23 02:08:39 46515:46515 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5675703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5676437Z STAGE:2022-11-23 02:08:40 46514:46514 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5677064Z STAGE:2022-11-23 02:08:40 46514:46514 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5677696Z STAGE:2022-11-23 02:08:40 46514:46514 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5678334Z STAGE:2022-11-23 02:08:40 46515:46515 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5678967Z STAGE:2022-11-23 02:08:40 46515:46515 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5679593Z STAGE:2022-11-23 02:08:40 46514:46514 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5680205Z STAGE:2022-11-23 02:08:40 46515:46515 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5680815Z STAGE:2022-11-23 02:08:40 46515:46515 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5681451Z STAGE:2022-11-23 02:08:40 46515:46515 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5682086Z STAGE:2022-11-23 02:08:40 46514:46514 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5682701Z STAGE:2022-11-23 02:08:40 46514:46514 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5683055Z ok (3.942s) 2022-11-23T02:32:19.5683192Z 2022-11-23T02:32:19.5683476Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5683804Z Ran 1 test in 3.943s 2022-11-23T02:32:19.5683959Z 2022-11-23T02:32:19.5684037Z OK 2022-11-23T02:32:19.5684160Z 2022-11-23T02:32:19.5684270Z Generating XML reports... 2022-11-23T02:32:19.5684917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020837.xml 2022-11-23T02:32:19.5685611Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5686278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5686735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5687340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5687820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5688047Z 2022-11-23T02:32:19.5688142Z Running tests... 2022-11-23T02:32:19.5688567Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5689102Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s) 2022-11-23T02:32:19.5689404Z 2022-11-23T02:32:19.5689683Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5690005Z Ran 1 test in 0.002s 2022-11-23T02:32:19.5690154Z 2022-11-23T02:32:19.5690246Z OK (skipped=1) 2022-11-23T02:32:19.5690395Z 2022-11-23T02:32:19.5690506Z Generating XML reports... 2022-11-23T02:32:19.5691152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020844.xml 2022-11-23T02:32:19.5691855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5692505Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5692958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5693571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5694058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5694281Z 2022-11-23T02:32:19.5694374Z Running tests... 2022-11-23T02:32:19.5694796Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5695352Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-11-23T02:32:19.5695659Z 2022-11-23T02:32:19.5695938Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5696248Z Ran 1 test in 0.002s 2022-11-23T02:32:19.5696401Z 2022-11-23T02:32:19.5696498Z OK (skipped=1) 2022-11-23T02:32:19.5696640Z 2022-11-23T02:32:19.5696751Z Generating XML reports... 2022-11-23T02:32:19.5697395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020847.xml 2022-11-23T02:32:19.5698092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5698745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5699198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5699803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5700287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5700511Z 2022-11-23T02:32:19.5700605Z Running tests... 2022-11-23T02:32:19.5701023Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5701581Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46819 2022-11-23T02:32:19.5702142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46820 2022-11-23T02:32:19.5702668Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5703368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5703820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5704431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5704908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5705367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5706078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5706785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5707279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5707887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5708350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5708815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5709511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5710055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5710530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5711028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.5711539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.5712237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5713006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5713673Z STAGE:2022-11-23 02:08:53 46819:46819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5714290Z STAGE:2022-11-23 02:08:53 46820:46820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5714910Z STAGE:2022-11-23 02:08:53 46820:46820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5715550Z STAGE:2022-11-23 02:08:53 46820:46820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5716255Z STAGE:2022-11-23 02:08:53 46819:46819 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5716890Z STAGE:2022-11-23 02:08:53 46819:46819 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5717525Z STAGE:2022-11-23 02:08:53 46820:46820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5718123Z STAGE:2022-11-23 02:08:53 46819:46819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5718744Z STAGE:2022-11-23 02:08:53 46819:46819 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5719376Z STAGE:2022-11-23 02:08:53 46819:46819 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5720003Z STAGE:2022-11-23 02:08:53 46820:46820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5720635Z STAGE:2022-11-23 02:08:53 46820:46820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5720995Z ok (3.940s) 2022-11-23T02:32:19.5721128Z 2022-11-23T02:32:19.5721413Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5721729Z Ran 1 test in 3.941s 2022-11-23T02:32:19.5721884Z 2022-11-23T02:32:19.5721961Z OK 2022-11-23T02:32:19.5722081Z 2022-11-23T02:32:19.5722195Z Generating XML reports... 2022-11-23T02:32:19.5722859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020850.xml 2022-11-23T02:32:19.5723558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5724217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5724674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5725284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5725814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5726037Z 2022-11-23T02:32:19.5726132Z Running tests... 2022-11-23T02:32:19.5726560Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5727114Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47014 2022-11-23T02:32:19.5727663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47015 2022-11-23T02:32:19.5728185Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5728876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5729323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5729932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5730412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5730875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5731590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5732042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5732655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5733128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5733581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5734278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5735020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5735555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5736041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5736446Z skip: Skipped due to small world size. (3.842s) 2022-11-23T02:32:19.5736641Z 2022-11-23T02:32:19.5736926Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5737242Z Ran 1 test in 3.842s 2022-11-23T02:32:19.5737389Z 2022-11-23T02:32:19.5737480Z OK (skipped=1) 2022-11-23T02:32:19.5737622Z 2022-11-23T02:32:19.5737732Z Generating XML reports... 2022-11-23T02:32:19.5738384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020857.xml 2022-11-23T02:32:19.5739088Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5739743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5740197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5740813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5741280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5741507Z 2022-11-23T02:32:19.5741599Z Running tests... 2022-11-23T02:32:19.5742025Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5742552Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2022-11-23T02:32:19.5742881Z 2022-11-23T02:32:19.5743211Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5743537Z Ran 1 test in 0.002s 2022-11-23T02:32:19.5743686Z 2022-11-23T02:32:19.5743784Z OK (skipped=1) 2022-11-23T02:32:19.5743922Z 2022-11-23T02:32:19.5744031Z Generating XML reports... 2022-11-23T02:32:19.5744682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020904.xml 2022-11-23T02:32:19.5745385Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5746035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5746494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5747107Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5747585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5747814Z 2022-11-23T02:32:19.5747899Z Running tests... 2022-11-23T02:32:19.5748316Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5748849Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2022-11-23T02:32:19.5749224Z 2022-11-23T02:32:19.5749507Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5749828Z Ran 1 test in 0.002s 2022-11-23T02:32:19.5749981Z 2022-11-23T02:32:19.5750073Z OK (skipped=1) 2022-11-23T02:32:19.5750219Z 2022-11-23T02:32:19.5750326Z Generating XML reports... 2022-11-23T02:32:19.5750963Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020907.xml 2022-11-23T02:32:19.5751663Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5752317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5752782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5753390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5753867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5754089Z 2022-11-23T02:32:19.5754181Z Running tests... 2022-11-23T02:32:19.5754602Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5755112Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-11-23T02:32:19.5755120Z 2022-11-23T02:32:19.5755401Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5755499Z Ran 1 test in 0.002s 2022-11-23T02:32:19.5755506Z 2022-11-23T02:32:19.5755597Z OK (skipped=1) 2022-11-23T02:32:19.5755606Z 2022-11-23T02:32:19.5755716Z Generating XML reports... 2022-11-23T02:32:19.5756274Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020910.xml 2022-11-23T02:32:19.5756609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5757005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5757174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5757578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5757761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5757768Z 2022-11-23T02:32:19.5757860Z Running tests... 2022-11-23T02:32:19.5758137Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5758497Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-11-23T02:32:19.5758504Z 2022-11-23T02:32:19.5758782Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5758878Z Ran 1 test in 0.002s 2022-11-23T02:32:19.5758888Z 2022-11-23T02:32:19.5758980Z OK (skipped=1) 2022-11-23T02:32:19.5758986Z 2022-11-23T02:32:19.5759099Z Generating XML reports... 2022-11-23T02:32:19.5759576Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020913.xml 2022-11-23T02:32:19.5759906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5760299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5760465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5760865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5761052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5761067Z 2022-11-23T02:32:19.5761152Z Running tests... 2022-11-23T02:32:19.5761512Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5761854Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47429 2022-11-23T02:32:19.5762073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47430 2022-11-23T02:32:19.5762345Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5762741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5762911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5763317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5763501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5763748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5764142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5764308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5764708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5764893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5765134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5765568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5765996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5766225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5766445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5766532Z ok (3.847s) 2022-11-23T02:32:19.5766539Z 2022-11-23T02:32:19.5766819Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5766908Z Ran 1 test in 3.847s 2022-11-23T02:32:19.5766921Z 2022-11-23T02:32:19.5766991Z OK 2022-11-23T02:32:19.5766997Z 2022-11-23T02:32:19.5767107Z Generating XML reports... 2022-11-23T02:32:19.5767595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020915.xml 2022-11-23T02:32:19.5767975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5768373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5768545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5768948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5769134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5769141Z 2022-11-23T02:32:19.5769233Z Running tests... 2022-11-23T02:32:19.5769506Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5769843Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47612 2022-11-23T02:32:19.5770062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47613 2022-11-23T02:32:19.5770337Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5770774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5770944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5771352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5771533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5771772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5772168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5772336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5772740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5772927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5773161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5773590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5774020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5774246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5774475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5774715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.5774954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.5775386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5775805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5776046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:32:19.5776276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:32:19.5776696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:32:19.5777167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:32:19.5777401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T02:32:19.5777643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T02:32:19.5778059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:32:19.5778480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:32:19.5778564Z ok (4.240s) 2022-11-23T02:32:19.5778571Z 2022-11-23T02:32:19.5778857Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5778954Z Ran 1 test in 4.241s 2022-11-23T02:32:19.5778961Z 2022-11-23T02:32:19.5779036Z OK 2022-11-23T02:32:19.5779045Z 2022-11-23T02:32:19.5779154Z Generating XML reports... 2022-11-23T02:32:19.5779642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020922.xml 2022-11-23T02:32:19.5779975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5780414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5780577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5780984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5781166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5781173Z 2022-11-23T02:32:19.5781267Z Running tests... 2022-11-23T02:32:19.5781547Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5781815Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s) 2022-11-23T02:32:19.5781822Z 2022-11-23T02:32:19.5782092Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5782190Z Ran 1 test in 0.002s 2022-11-23T02:32:19.5782197Z 2022-11-23T02:32:19.5782289Z OK (skipped=1) 2022-11-23T02:32:19.5782298Z 2022-11-23T02:32:19.5782407Z Generating XML reports... 2022-11-23T02:32:19.5782889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020930.xml 2022-11-23T02:32:19.5783215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5783608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5783774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5784175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5784362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5784368Z 2022-11-23T02:32:19.5784459Z Running tests... 2022-11-23T02:32:19.5784735Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5785094Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47877 2022-11-23T02:32:19.5785310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47878 2022-11-23T02:32:19.5785577Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5785971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5786136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5786588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5786765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5787018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5787447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5787840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5788004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5788407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5788588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5788834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5789257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5789563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5789788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5790028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.5790265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.5790689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5791114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5791480Z STAGE:2022-11-23 02:09:35 47877:47877 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5791839Z STAGE:2022-11-23 02:09:35 47878:47878 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5792204Z STAGE:2022-11-23 02:09:35 47877:47877 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5792556Z STAGE:2022-11-23 02:09:35 47878:47878 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5792930Z STAGE:2022-11-23 02:09:35 47877:47877 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5793299Z STAGE:2022-11-23 02:09:35 47878:47878 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5793650Z STAGE:2022-11-23 02:09:35 47877:47877 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5794000Z STAGE:2022-11-23 02:09:35 47878:47878 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5794355Z STAGE:2022-11-23 02:09:35 47878:47878 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5794718Z STAGE:2022-11-23 02:09:35 47878:47878 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5795079Z STAGE:2022-11-23 02:09:35 47877:47877 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5795447Z STAGE:2022-11-23 02:09:35 47877:47877 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5796332Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5796435Z warnings.warn( 2022-11-23T02:32:19.5797308Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5797406Z warnings.warn( 2022-11-23T02:32:19.5797493Z ok (4.042s) 2022-11-23T02:32:19.5797500Z 2022-11-23T02:32:19.5797785Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5797885Z Ran 1 test in 4.043s 2022-11-23T02:32:19.5797892Z 2022-11-23T02:32:19.5797968Z OK 2022-11-23T02:32:19.5797974Z 2022-11-23T02:32:19.5798082Z Generating XML reports... 2022-11-23T02:32:19.5798568Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020933.xml 2022-11-23T02:32:19.5798899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5799294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5799464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5799865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5800099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5800108Z 2022-11-23T02:32:19.5800203Z Running tests... 2022-11-23T02:32:19.5800481Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5800830Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48072 2022-11-23T02:32:19.5801044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48073 2022-11-23T02:32:19.5801316Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5801715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5801882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5802285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5802471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5802709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5803101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5803269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5803675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5803862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5804103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5804537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5804962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5805186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5805406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5805644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.5805885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.5806363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5806783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5807151Z STAGE:2022-11-23 02:09:42 48072:48072 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5807496Z STAGE:2022-11-23 02:09:42 48073:48073 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5807856Z STAGE:2022-11-23 02:09:42 48072:48072 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5808229Z STAGE:2022-11-23 02:09:42 48072:48072 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5808586Z STAGE:2022-11-23 02:09:42 48073:48073 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5808966Z STAGE:2022-11-23 02:09:42 48073:48073 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5809315Z STAGE:2022-11-23 02:09:42 48072:48072 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5809661Z STAGE:2022-11-23 02:09:42 48073:48073 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5810060Z STAGE:2022-11-23 02:09:42 48073:48073 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5810436Z STAGE:2022-11-23 02:09:42 48073:48073 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5810786Z STAGE:2022-11-23 02:09:42 48072:48072 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5811155Z STAGE:2022-11-23 02:09:42 48072:48072 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5811967Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5812073Z warnings.warn( 2022-11-23T02:32:19.5812891Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5812985Z warnings.warn( 2022-11-23T02:32:19.5813069Z ok (4.042s) 2022-11-23T02:32:19.5813076Z 2022-11-23T02:32:19.5813358Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5813459Z Ran 1 test in 4.043s 2022-11-23T02:32:19.5813467Z 2022-11-23T02:32:19.5813545Z OK 2022-11-23T02:32:19.5813550Z 2022-11-23T02:32:19.5813660Z Generating XML reports... 2022-11-23T02:32:19.5814148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020940.xml 2022-11-23T02:32:19.5814484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5814881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5815055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5815463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5815649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5815656Z 2022-11-23T02:32:19.5815750Z Running tests... 2022-11-23T02:32:19.5816031Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5816387Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48267 2022-11-23T02:32:19.5816651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48268 2022-11-23T02:32:19.5816924Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5817321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5817488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5817895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5818070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5818313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5818702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5818871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5819274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5819502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5819745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5820174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5820601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5820827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5821042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5821281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.5821518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.5821944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5822366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5822729Z STAGE:2022-11-23 02:09:49 48267:48267 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5823076Z STAGE:2022-11-23 02:09:49 48268:48268 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5823437Z STAGE:2022-11-23 02:09:49 48267:48267 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5823818Z STAGE:2022-11-23 02:09:49 48267:48267 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5824178Z STAGE:2022-11-23 02:09:49 48268:48268 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5824556Z STAGE:2022-11-23 02:09:49 48268:48268 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5824901Z STAGE:2022-11-23 02:09:49 48267:48267 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5825244Z STAGE:2022-11-23 02:09:49 48268:48268 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5825597Z STAGE:2022-11-23 02:09:49 48268:48268 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5825959Z STAGE:2022-11-23 02:09:49 48268:48268 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5826316Z STAGE:2022-11-23 02:09:49 48267:48267 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5826736Z STAGE:2022-11-23 02:09:49 48267:48267 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5827554Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5827652Z warnings.warn( 2022-11-23T02:32:19.5828462Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5828559Z warnings.warn( 2022-11-23T02:32:19.5828645Z ok (3.849s) 2022-11-23T02:32:19.5828652Z 2022-11-23T02:32:19.5828934Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5829035Z Ran 1 test in 3.849s 2022-11-23T02:32:19.5829042Z 2022-11-23T02:32:19.5829118Z OK 2022-11-23T02:32:19.5829124Z 2022-11-23T02:32:19.5829235Z Generating XML reports... 2022-11-23T02:32:19.5829725Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020947.xml 2022-11-23T02:32:19.5830098Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5830497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5830664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5831071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5831251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5831258Z 2022-11-23T02:32:19.5831354Z Running tests... 2022-11-23T02:32:19.5831632Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5831982Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48462 2022-11-23T02:32:19.5832201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48463 2022-11-23T02:32:19.5832470Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5832863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5833027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5833424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5833604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5833852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5834243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5834411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5834814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5834999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5835245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5835677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5836177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5836576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5836796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5837041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.5837278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.5837711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5838134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5838496Z STAGE:2022-11-23 02:09:56 48462:48462 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5838849Z STAGE:2022-11-23 02:09:56 48463:48463 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5839207Z STAGE:2022-11-23 02:09:56 48462:48462 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5839630Z STAGE:2022-11-23 02:09:56 48462:48462 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5839990Z STAGE:2022-11-23 02:09:56 48463:48463 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5840365Z STAGE:2022-11-23 02:09:56 48463:48463 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5840719Z STAGE:2022-11-23 02:09:56 48462:48462 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5841067Z STAGE:2022-11-23 02:09:56 48463:48463 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5841417Z STAGE:2022-11-23 02:09:56 48463:48463 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5841792Z STAGE:2022-11-23 02:09:56 48463:48463 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5842146Z STAGE:2022-11-23 02:09:56 48462:48462 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5842519Z STAGE:2022-11-23 02:09:56 48462:48462 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5843333Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5843434Z warnings.warn( 2022-11-23T02:32:19.5844239Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5844338Z warnings.warn( 2022-11-23T02:32:19.5844425Z ok (3.940s) 2022-11-23T02:32:19.5844433Z 2022-11-23T02:32:19.5844717Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5844816Z Ran 1 test in 3.941s 2022-11-23T02:32:19.5844824Z 2022-11-23T02:32:19.5844903Z OK 2022-11-23T02:32:19.5844910Z 2022-11-23T02:32:19.5845019Z Generating XML reports... 2022-11-23T02:32:19.5845506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020954.xml 2022-11-23T02:32:19.5845839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5846237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5846406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5846863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5847049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5847057Z 2022-11-23T02:32:19.5847147Z Running tests... 2022-11-23T02:32:19.5847433Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5847783Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48657 2022-11-23T02:32:19.5848001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48658 2022-11-23T02:32:19.5848270Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5848664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5848827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5849231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5849414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5849704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5850100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5850264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5850666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5850850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5851096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5851528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5851952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5852182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5852401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5852551Z skip: Skipped due to small world size. (3.849s) 2022-11-23T02:32:19.5852558Z 2022-11-23T02:32:19.5852837Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5852936Z Ran 1 test in 3.850s 2022-11-23T02:32:19.5852943Z 2022-11-23T02:32:19.5853034Z OK (skipped=1) 2022-11-23T02:32:19.5853040Z 2022-11-23T02:32:19.5853149Z Generating XML reports... 2022-11-23T02:32:19.5853636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021001.xml 2022-11-23T02:32:19.5853969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5854363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5854531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5854925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5855107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5855113Z 2022-11-23T02:32:19.5855209Z Running tests... 2022-11-23T02:32:19.5855488Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5855831Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48840 2022-11-23T02:32:19.5856094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48841 2022-11-23T02:32:19.5856366Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5856764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5856933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5857337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5857519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5857762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5858157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5858326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5858730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5858953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5859199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5859630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5860051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5860278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5860500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5860654Z skip: Skipped due to small world size. (3.844s) 2022-11-23T02:32:19.5860661Z 2022-11-23T02:32:19.5860947Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5861038Z Ran 1 test in 3.845s 2022-11-23T02:32:19.5861055Z 2022-11-23T02:32:19.5861144Z OK (skipped=1) 2022-11-23T02:32:19.5861151Z 2022-11-23T02:32:19.5861263Z Generating XML reports... 2022-11-23T02:32:19.5861756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021008.xml 2022-11-23T02:32:19.5862086Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5862491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5862660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5863066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5863253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5863260Z 2022-11-23T02:32:19.5863352Z Running tests... 2022-11-23T02:32:19.5863634Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5863988Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49023 2022-11-23T02:32:19.5864204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49024 2022-11-23T02:32:19.5864474Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5864862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5865073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5865477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5865663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5865912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5866301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5866469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5866875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5867056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5867291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5867726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5868149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5868417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5868636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5868788Z skip: Skipped due to small world size. (3.739s) 2022-11-23T02:32:19.5868794Z 2022-11-23T02:32:19.5869080Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5869177Z Ran 1 test in 3.740s 2022-11-23T02:32:19.5869184Z 2022-11-23T02:32:19.5869276Z OK (skipped=1) 2022-11-23T02:32:19.5869282Z 2022-11-23T02:32:19.5869395Z Generating XML reports... 2022-11-23T02:32:19.5869885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021015.xml 2022-11-23T02:32:19.5870219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5870613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5870779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5871180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5871361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5871368Z 2022-11-23T02:32:19.5871459Z Running tests... 2022-11-23T02:32:19.5871743Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5872090Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49206 2022-11-23T02:32:19.5872311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49207 2022-11-23T02:32:19.5872586Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5872987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5873154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5873548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5873733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5873978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5874369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5874585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5874990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5875174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5875414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5875847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5876337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5876566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5876791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5876942Z skip: Skipped due to small world size. (3.741s) 2022-11-23T02:32:19.5876949Z 2022-11-23T02:32:19.5877231Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5877327Z Ran 1 test in 3.741s 2022-11-23T02:32:19.5877334Z 2022-11-23T02:32:19.5877473Z OK (skipped=1) 2022-11-23T02:32:19.5877481Z 2022-11-23T02:32:19.5877591Z Generating XML reports... 2022-11-23T02:32:19.5878081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021021.xml 2022-11-23T02:32:19.5878410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5878805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5878974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5879381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5879558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5879571Z 2022-11-23T02:32:19.5879656Z Running tests... 2022-11-23T02:32:19.5879939Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5880280Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49389 2022-11-23T02:32:19.5880494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49390 2022-11-23T02:32:19.5880765Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5881155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5881323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5881732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5881911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5882156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5882543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5882709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5883112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5883293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5883534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5884018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5884442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5884673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5885035Z STAGE:2022-11-23 02:10:31 49390:49390 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5885258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5885619Z STAGE:2022-11-23 02:10:31 49389:49389 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5885984Z STAGE:2022-11-23 02:10:31 49390:49390 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5886340Z STAGE:2022-11-23 02:10:31 49389:49389 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5886711Z STAGE:2022-11-23 02:10:31 49390:49390 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5887085Z STAGE:2022-11-23 02:10:31 49389:49389 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5887480Z STAGE:2022-11-23 02:10:31 49389:49389 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5887830Z STAGE:2022-11-23 02:10:31 49390:49390 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5888189Z STAGE:2022-11-23 02:10:31 49389:49389 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5888562Z STAGE:2022-11-23 02:10:31 49389:49389 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5888916Z STAGE:2022-11-23 02:10:31 49390:49390 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5889284Z STAGE:2022-11-23 02:10:31 49390:49390 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5890110Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5890210Z warnings.warn( 2022-11-23T02:32:19.5891021Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5891119Z warnings.warn( 2022-11-23T02:32:19.5891205Z ok (3.851s) 2022-11-23T02:32:19.5891213Z 2022-11-23T02:32:19.5891492Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5891591Z Ran 1 test in 3.852s 2022-11-23T02:32:19.5891601Z 2022-11-23T02:32:19.5891681Z OK 2022-11-23T02:32:19.5891688Z 2022-11-23T02:32:19.5891800Z Generating XML reports... 2022-11-23T02:32:19.5892287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021028.xml 2022-11-23T02:32:19.5892628Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5893027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5893197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5893606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5893791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5893798Z 2022-11-23T02:32:19.5893894Z Running tests... 2022-11-23T02:32:19.5894216Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5894588Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49578 2022-11-23T02:32:19.5894811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49579 2022-11-23T02:32:19.5895086Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5895480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5895647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5896054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5896237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5896481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5896914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5897344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5897514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5897921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5898102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5898343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5898767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5899002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5899225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5900037Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5900137Z warnings.warn( 2022-11-23T02:32:19.5900947Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5901043Z warnings.warn( 2022-11-23T02:32:19.5901131Z ok (3.839s) 2022-11-23T02:32:19.5901139Z 2022-11-23T02:32:19.5901422Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5901520Z Ran 1 test in 3.839s 2022-11-23T02:32:19.5901526Z 2022-11-23T02:32:19.5901594Z OK 2022-11-23T02:32:19.5901609Z 2022-11-23T02:32:19.5901712Z Generating XML reports... 2022-11-23T02:32:19.5902200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021035.xml 2022-11-23T02:32:19.5902533Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5902928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5903094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5903499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5903734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5903741Z 2022-11-23T02:32:19.5903833Z Running tests... 2022-11-23T02:32:19.5904113Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5904450Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49761 2022-11-23T02:32:19.5904665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49762 2022-11-23T02:32:19.5904936Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5905324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5905494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5905898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5906085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5906324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5906781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5906952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5907359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5907545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5907853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5908275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5908751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5909142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5909558Z STAGE:2022-11-23 02:10:44 49762:49762 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5909825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5910229Z STAGE:2022-11-23 02:10:44 49761:49761 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5910648Z STAGE:2022-11-23 02:10:44 49761:49761 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5911067Z STAGE:2022-11-23 02:10:44 49761:49761 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5911476Z STAGE:2022-11-23 02:10:44 49762:49762 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5911844Z STAGE:2022-11-23 02:10:44 49762:49762 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5912241Z STAGE:2022-11-23 02:10:44 49761:49761 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5912662Z STAGE:2022-11-23 02:10:44 49762:49762 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5913066Z STAGE:2022-11-23 02:10:44 49761:49761 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5913486Z STAGE:2022-11-23 02:10:44 49761:49761 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5913886Z STAGE:2022-11-23 02:10:44 49762:49762 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5914309Z STAGE:2022-11-23 02:10:44 49762:49762 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5931178Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5931529Z warnings.warn( 2022-11-23T02:32:19.5932498Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5932634Z warnings.warn( 2022-11-23T02:32:19.5932741Z ok (3.856s) 2022-11-23T02:32:19.5932751Z 2022-11-23T02:32:19.5933061Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5933164Z Ran 1 test in 3.857s 2022-11-23T02:32:19.5933172Z 2022-11-23T02:32:19.5933251Z OK 2022-11-23T02:32:19.5933257Z 2022-11-23T02:32:19.5933377Z Generating XML reports... 2022-11-23T02:32:19.5933879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021042.xml 2022-11-23T02:32:19.5934214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5934685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5934865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5935275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5935466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5935473Z 2022-11-23T02:32:19.5935569Z Running tests... 2022-11-23T02:32:19.5935856Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5936211Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49950 2022-11-23T02:32:19.5936437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49951 2022-11-23T02:32:19.5936715Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5937117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5937290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5937698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5937889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5938137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5938538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5938708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5939114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5939306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5939552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5939990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5940422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5940656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5941075Z STAGE:2022-11-23 02:10:51 49951:49951 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5941306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5941665Z STAGE:2022-11-23 02:10:51 49950:49950 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5942025Z STAGE:2022-11-23 02:10:51 49950:49950 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5942377Z STAGE:2022-11-23 02:10:51 49951:49951 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5942842Z STAGE:2022-11-23 02:10:51 49950:49950 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5943125Z STAGE:2022-11-23 02:10:51 49951:49951 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5943386Z STAGE:2022-11-23 02:10:51 49950:49950 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5943633Z STAGE:2022-11-23 02:10:51 49951:49951 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5943888Z STAGE:2022-11-23 02:10:51 49950:49950 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5944188Z STAGE:2022-11-23 02:10:51 49950:49950 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5944443Z STAGE:2022-11-23 02:10:51 49951:49951 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5944707Z STAGE:2022-11-23 02:10:51 49951:49951 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5945283Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5945361Z warnings.warn( 2022-11-23T02:32:19.5945948Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5946020Z warnings.warn( 2022-11-23T02:32:19.5946086Z ok (3.948s) 2022-11-23T02:32:19.5946092Z 2022-11-23T02:32:19.5946297Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5946370Z Ran 1 test in 3.949s 2022-11-23T02:32:19.5946376Z 2022-11-23T02:32:19.5946435Z OK 2022-11-23T02:32:19.5946440Z 2022-11-23T02:32:19.5946522Z Generating XML reports... 2022-11-23T02:32:19.5946870Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021049.xml 2022-11-23T02:32:19.5947114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5947402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5947527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5947821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5947956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5947961Z 2022-11-23T02:32:19.5948024Z Running tests... 2022-11-23T02:32:19.5948226Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5948474Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50139 2022-11-23T02:32:19.5948630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50140 2022-11-23T02:32:19.5948829Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5949148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5949271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5949560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5949693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5949870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5950149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5950268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5950552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5950688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5950861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5951198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5951503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5951666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5951824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5952085Z STAGE:2022-11-23 02:10:58 50139:50139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5952337Z STAGE:2022-11-23 02:10:58 50140:50140 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5952597Z STAGE:2022-11-23 02:10:58 50140:50140 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5952863Z STAGE:2022-11-23 02:10:58 50140:50140 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5953117Z STAGE:2022-11-23 02:10:58 50139:50139 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5953388Z STAGE:2022-11-23 02:10:58 50139:50139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5953638Z STAGE:2022-11-23 02:10:58 50139:50139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5953884Z STAGE:2022-11-23 02:10:58 50140:50140 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5954138Z STAGE:2022-11-23 02:10:58 50140:50140 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5954393Z STAGE:2022-11-23 02:10:58 50139:50139 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5954662Z STAGE:2022-11-23 02:10:58 50140:50140 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5954923Z STAGE:2022-11-23 02:10:58 50139:50139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5955503Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5955579Z warnings.warn( 2022-11-23T02:32:19.5956386Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:32:19.5956507Z warnings.warn( 2022-11-23T02:32:19.5956574Z ok (3.839s) 2022-11-23T02:32:19.5956580Z 2022-11-23T02:32:19.5956797Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5956878Z Ran 1 test in 3.839s 2022-11-23T02:32:19.5956884Z 2022-11-23T02:32:19.5956944Z OK 2022-11-23T02:32:19.5956949Z 2022-11-23T02:32:19.5957037Z Generating XML reports... 2022-11-23T02:32:19.5957406Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021056.xml 2022-11-23T02:32:19.5957658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5957959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5958091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5958399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5958544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5958549Z 2022-11-23T02:32:19.5958624Z Running tests... 2022-11-23T02:32:19.5958840Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5959137Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50328 2022-11-23T02:32:19.5959304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50329 2022-11-23T02:32:19.5959512Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5959811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5959942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5960247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5960390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5960577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5960877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5961007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5961311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5961452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5961637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5961962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5962283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5962464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5962637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5962710Z ok (3.839s) 2022-11-23T02:32:19.5962715Z 2022-11-23T02:32:19.5962939Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5963017Z Ran 1 test in 3.840s 2022-11-23T02:32:19.5963022Z 2022-11-23T02:32:19.5963087Z OK 2022-11-23T02:32:19.5963091Z 2022-11-23T02:32:19.5963182Z Generating XML reports... 2022-11-23T02:32:19.5963543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021103.xml 2022-11-23T02:32:19.5963843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5964142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5964274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5964577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5964722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5964728Z 2022-11-23T02:32:19.5964807Z Running tests... 2022-11-23T02:32:19.5965017Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5965272Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50511 2022-11-23T02:32:19.5965436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50512 2022-11-23T02:32:19.5965643Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5965970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5966100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5966403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5987625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5987976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.5988537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5988683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5989009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5989149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5989339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5989672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5989992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.5990163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.5990327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.5990510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.5990696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.5991016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5991335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.5991613Z STAGE:2022-11-23 02:11:12 50511:50511 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5991878Z STAGE:2022-11-23 02:11:12 50512:50512 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5992155Z STAGE:2022-11-23 02:11:12 50511:50511 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5992420Z STAGE:2022-11-23 02:11:12 50512:50512 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5992870Z STAGE:2022-11-23 02:11:12 50511:50511 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5993152Z STAGE:2022-11-23 02:11:12 50512:50512 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5993418Z STAGE:2022-11-23 02:11:12 50511:50511 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5993681Z STAGE:2022-11-23 02:11:12 50512:50512 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.5993948Z STAGE:2022-11-23 02:11:12 50512:50512 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5994221Z STAGE:2022-11-23 02:11:12 50512:50512 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5994485Z STAGE:2022-11-23 02:11:12 50511:50511 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.5994761Z STAGE:2022-11-23 02:11:12 50511:50511 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.5994827Z ok (4.142s) 2022-11-23T02:32:19.5994835Z 2022-11-23T02:32:19.5995047Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5995121Z Ran 1 test in 4.143s 2022-11-23T02:32:19.5995127Z 2022-11-23T02:32:19.5995183Z OK 2022-11-23T02:32:19.5995188Z 2022-11-23T02:32:19.5995323Z Generating XML reports... 2022-11-23T02:32:19.5995695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021110.xml 2022-11-23T02:32:19.5995940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.5996493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5996623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5996936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5997082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5997088Z 2022-11-23T02:32:19.5997158Z Running tests... 2022-11-23T02:32:19.5997368Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.5997629Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50706 2022-11-23T02:32:19.5997793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50707 2022-11-23T02:32:19.5998002Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.5998302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5998429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5998741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5998882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.5999073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.5999381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.5999509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.5999815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.5999955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6000142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6000521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6000847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6001023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6001185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6001366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6001548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6001875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6002199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6002480Z STAGE:2022-11-23 02:11:19 50706:50706 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6002745Z STAGE:2022-11-23 02:11:19 50707:50707 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6003155Z STAGE:2022-11-23 02:11:19 50706:50706 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6003426Z STAGE:2022-11-23 02:11:19 50707:50707 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6003715Z STAGE:2022-11-23 02:11:19 50706:50706 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6004002Z STAGE:2022-11-23 02:11:19 50707:50707 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6004271Z STAGE:2022-11-23 02:11:19 50706:50706 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6004533Z STAGE:2022-11-23 02:11:19 50707:50707 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6004811Z STAGE:2022-11-23 02:11:19 50707:50707 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6005077Z STAGE:2022-11-23 02:11:19 50706:50706 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6005365Z STAGE:2022-11-23 02:11:19 50707:50707 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6005646Z STAGE:2022-11-23 02:11:19 50706:50706 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6005713Z ok (3.739s) 2022-11-23T02:32:19.6005719Z 2022-11-23T02:32:19.6005934Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6006007Z Ran 1 test in 3.739s 2022-11-23T02:32:19.6006013Z 2022-11-23T02:32:19.6006071Z OK 2022-11-23T02:32:19.6006075Z 2022-11-23T02:32:19.6006158Z Generating XML reports... 2022-11-23T02:32:19.6006535Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021117.xml 2022-11-23T02:32:19.6006788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6007196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6007322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6007627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6007763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6007768Z 2022-11-23T02:32:19.6007836Z Running tests... 2022-11-23T02:32:19.6008047Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6008301Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50901 2022-11-23T02:32:19.6008532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50902 2022-11-23T02:32:19.6008737Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6009034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6009157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6009460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6009596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6009778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6010065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6010191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6010487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6010657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6010840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6011157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6011471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6011639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6011800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6011981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6012157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6012479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6012748Z STAGE:2022-11-23 02:11:26 50901:50901 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6013061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6013324Z STAGE:2022-11-23 02:11:26 50902:50902 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6013610Z STAGE:2022-11-23 02:11:26 50901:50901 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6013872Z STAGE:2022-11-23 02:11:26 50902:50902 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6014148Z STAGE:2022-11-23 02:11:26 50901:50901 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6014424Z STAGE:2022-11-23 02:11:26 50902:50902 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6014683Z STAGE:2022-11-23 02:11:26 50901:50901 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6014937Z STAGE:2022-11-23 02:11:26 50902:50902 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6015200Z STAGE:2022-11-23 02:11:26 50901:50901 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6015472Z STAGE:2022-11-23 02:11:26 50901:50901 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6015736Z STAGE:2022-11-23 02:11:26 50902:50902 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6016045Z STAGE:2022-11-23 02:11:26 50902:50902 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6016108Z ok (4.140s) 2022-11-23T02:32:19.6016114Z 2022-11-23T02:32:19.6016322Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6016394Z Ran 1 test in 4.141s 2022-11-23T02:32:19.6016402Z 2022-11-23T02:32:19.6016459Z OK 2022-11-23T02:32:19.6016463Z 2022-11-23T02:32:19.6016544Z Generating XML reports... 2022-11-23T02:32:19.6016910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021124.xml 2022-11-23T02:32:19.6017156Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6017445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6017585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6017887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6018024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6018030Z 2022-11-23T02:32:19.6018097Z Running tests... 2022-11-23T02:32:19.6018335Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6018584Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51096 2022-11-23T02:32:19.6018745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51097 2022-11-23T02:32:19.6018944Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6019237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6019362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6019659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6019794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6019978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6020266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6020389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6020690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6020825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6041679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6042160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6042496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6042672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6042842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6043018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6043196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6043514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6043931Z STAGE:2022-11-23 02:11:33 51096:51096 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6044245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6044512Z STAGE:2022-11-23 02:11:34 51097:51097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6044785Z STAGE:2022-11-23 02:11:34 51096:51096 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6045048Z STAGE:2022-11-23 02:11:34 51097:51097 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6045330Z STAGE:2022-11-23 02:11:34 51096:51096 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6045590Z STAGE:2022-11-23 02:11:34 51096:51096 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6045868Z STAGE:2022-11-23 02:11:34 51097:51097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6046133Z STAGE:2022-11-23 02:11:34 51097:51097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6046396Z STAGE:2022-11-23 02:11:34 51097:51097 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6046918Z STAGE:2022-11-23 02:11:34 51096:51096 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:11:34 51097:51097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6046928Z 2022-11-23T02:32:19.6047214Z STAGE:2022-11-23 02:11:34 51096:51096 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6047274Z ok (4.049s) 2022-11-23T02:32:19.6047289Z 2022-11-23T02:32:19.6047496Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6047579Z Ran 1 test in 4.049s 2022-11-23T02:32:19.6047585Z 2022-11-23T02:32:19.6047647Z OK 2022-11-23T02:32:19.6047652Z 2022-11-23T02:32:19.6047744Z Generating XML reports... 2022-11-23T02:32:19.6048120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021131.xml 2022-11-23T02:32:19.6048376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6048682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6048812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6049120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6049261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6049267Z 2022-11-23T02:32:19.6049340Z Running tests... 2022-11-23T02:32:19.6049549Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6049800Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51291 2022-11-23T02:32:19.6049967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51292 2022-11-23T02:32:19.6050175Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6050475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6050605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6050912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6051053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6051241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6051616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6051907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6052029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6052336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6052474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6052658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6052977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6053152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6053323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6053438Z skip: Skipped due to small world size. (3.739s) 2022-11-23T02:32:19.6053444Z 2022-11-23T02:32:19.6053660Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6053773Z Ran 1 test in 3.740s 2022-11-23T02:32:19.6053779Z 2022-11-23T02:32:19.6053853Z OK (skipped=1) 2022-11-23T02:32:19.6053857Z 2022-11-23T02:32:19.6053941Z Generating XML reports... 2022-11-23T02:32:19.6054311Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021138.xml 2022-11-23T02:32:19.6054562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6054859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6054988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6055294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6055435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6055441Z 2022-11-23T02:32:19.6055512Z Running tests... 2022-11-23T02:32:19.6055729Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6055977Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51474 2022-11-23T02:32:19.6056140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51475 2022-11-23T02:32:19.6056334Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6056637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6056769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6057072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6057209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6057399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6057692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6057819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6058126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6058266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6058448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6058812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6059132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6059305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6059476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6059588Z skip: Skipped due to small world size. (3.759s) 2022-11-23T02:32:19.6059593Z 2022-11-23T02:32:19.6059807Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6059883Z Ran 1 test in 3.759s 2022-11-23T02:32:19.6059888Z 2022-11-23T02:32:19.6059960Z OK (skipped=1) 2022-11-23T02:32:19.6059966Z 2022-11-23T02:32:19.6060052Z Generating XML reports... 2022-11-23T02:32:19.6060421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021145.xml 2022-11-23T02:32:19.6060725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6061080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6061203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6061510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6061651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6061656Z 2022-11-23T02:32:19.6061730Z Running tests... 2022-11-23T02:32:19.6061945Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6062194Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51657 2022-11-23T02:32:19.6062362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51658 2022-11-23T02:32:19.6062721Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6063017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6063144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6063449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6063587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6063770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6064070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6064197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6064500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6064642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6064828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6065152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6065470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6065647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6065851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6065958Z skip: Skipped due to small world size. (3.639s) 2022-11-23T02:32:19.6065973Z 2022-11-23T02:32:19.6066177Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6066253Z Ran 1 test in 3.640s 2022-11-23T02:32:19.6066260Z 2022-11-23T02:32:19.6066330Z OK (skipped=1) 2022-11-23T02:32:19.6066335Z 2022-11-23T02:32:19.6066420Z Generating XML reports... 2022-11-23T02:32:19.6066783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021152.xml 2022-11-23T02:32:19.6067032Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6067326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6067451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6067755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6067893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6067898Z 2022-11-23T02:32:19.6067968Z Running tests... 2022-11-23T02:32:19.6068225Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6068473Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51840 2022-11-23T02:32:19.6068637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51841 2022-11-23T02:32:19.6068841Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6069134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6069265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6069570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6069711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6069899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6070223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6070517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6070635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6070936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6071075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6071258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6071584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6071756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6071929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6072042Z skip: Skipped due to small world size. (3.949s) 2022-11-23T02:32:19.6072047Z 2022-11-23T02:32:19.6072260Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6072334Z Ran 1 test in 3.950s 2022-11-23T02:32:19.6072340Z 2022-11-23T02:32:19.6072412Z OK (skipped=1) 2022-11-23T02:32:19.6072416Z 2022-11-23T02:32:19.6072500Z Generating XML reports... 2022-11-23T02:32:19.6072907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021158.xml 2022-11-23T02:32:19.6094335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6094835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6094998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6095371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6095537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6095546Z 2022-11-23T02:32:19.6095631Z Running tests... 2022-11-23T02:32:19.6095880Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6096163Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52023 2022-11-23T02:32:19.6096357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52024 2022-11-23T02:32:19.6096594Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6097091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6097244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6097597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6097759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6097976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6098311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6098459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6098803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6098967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6099176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6099541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6099905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6100102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6100416Z STAGE:2022-11-23 02:12:08 52024:52024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6100607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6100905Z STAGE:2022-11-23 02:12:08 52023:52023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6101211Z STAGE:2022-11-23 02:12:08 52023:52023 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6101508Z STAGE:2022-11-23 02:12:08 52024:52024 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6101828Z STAGE:2022-11-23 02:12:08 52023:52023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6102144Z STAGE:2022-11-23 02:12:08 52024:52024 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6102445Z STAGE:2022-11-23 02:12:08 52023:52023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6102815Z STAGE:2022-11-23 02:12:08 52024:52024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6103122Z STAGE:2022-11-23 02:12:08 52023:52023 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6103406Z STAGE:2022-11-23 02:12:08 52024:52024 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6103728Z STAGE:2022-11-23 02:12:08 52023:52023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6104042Z STAGE:2022-11-23 02:12:08 52024:52024 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6104119Z ok (3.839s) 2022-11-23T02:32:19.6104125Z 2022-11-23T02:32:19.6104370Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6104455Z Ran 1 test in 3.839s 2022-11-23T02:32:19.6104461Z 2022-11-23T02:32:19.6104529Z OK 2022-11-23T02:32:19.6104534Z 2022-11-23T02:32:19.6104632Z Generating XML reports... 2022-11-23T02:32:19.6105052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021205.xml 2022-11-23T02:32:19.6105335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6105718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6105867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6106214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6106375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6106381Z 2022-11-23T02:32:19.6106463Z Running tests... 2022-11-23T02:32:19.6106702Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6106977Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52212 2022-11-23T02:32:19.6107168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52213 2022-11-23T02:32:19.6107403Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6107736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6107879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6108220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6108369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6108577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6108907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6109054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6109394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6109551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6109761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6110127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6110489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6110678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6110987Z STAGE:2022-11-23 02:12:15 52213:52213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6111225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6111526Z STAGE:2022-11-23 02:12:15 52212:52212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6111834Z STAGE:2022-11-23 02:12:15 52212:52212 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6112130Z STAGE:2022-11-23 02:12:15 52213:52213 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6112447Z STAGE:2022-11-23 02:12:15 52212:52212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6112762Z STAGE:2022-11-23 02:12:15 52213:52213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6113063Z STAGE:2022-11-23 02:12:15 52213:52213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6113357Z STAGE:2022-11-23 02:12:15 52212:52212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6113664Z STAGE:2022-11-23 02:12:15 52212:52212 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6113977Z STAGE:2022-11-23 02:12:15 52212:52212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6114319Z STAGE:2022-11-23 02:12:15 52213:52213 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6114631Z STAGE:2022-11-23 02:12:15 52213:52213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6114706Z ok (3.741s) 2022-11-23T02:32:19.6114712Z 2022-11-23T02:32:19.6114941Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6115031Z Ran 1 test in 3.742s 2022-11-23T02:32:19.6115036Z 2022-11-23T02:32:19.6115105Z OK 2022-11-23T02:32:19.6115111Z 2022-11-23T02:32:19.6115208Z Generating XML reports... 2022-11-23T02:32:19.6115625Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021212.xml 2022-11-23T02:32:19.6115915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6116336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6116483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6116827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6116990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6116996Z 2022-11-23T02:32:19.6117080Z Running tests... 2022-11-23T02:32:19.6117322Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6117599Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52401 2022-11-23T02:32:19.6117788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52402 2022-11-23T02:32:19.6118021Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6118354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6118499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6118844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6119005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6119214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6119546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6119742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6120077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6120239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6120450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6120819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6121180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6121373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6121568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6121881Z STAGE:2022-11-23 02:12:22 52401:52401 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6122179Z STAGE:2022-11-23 02:12:22 52402:52402 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6122530Z STAGE:2022-11-23 02:12:22 52402:52402 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6122855Z STAGE:2022-11-23 02:12:22 52402:52402 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6123060Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6123362Z STAGE:2022-11-23 02:12:22 52401:52401 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6123676Z STAGE:2022-11-23 02:12:22 52401:52401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6123874Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6124172Z STAGE:2022-11-23 02:12:22 52402:52402 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6124461Z STAGE:2022-11-23 02:12:22 52401:52401 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6124765Z STAGE:2022-11-23 02:12:22 52401:52401 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6125080Z STAGE:2022-11-23 02:12:22 52401:52401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6125386Z STAGE:2022-11-23 02:12:22 52402:52402 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6125702Z STAGE:2022-11-23 02:12:22 52402:52402 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6126434Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:32:19.6126526Z warnings.warn( 2022-11-23T02:32:19.6127244Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:32:19.6127330Z warnings.warn( 2022-11-23T02:32:19.6127405Z ok (4.242s) 2022-11-23T02:32:19.6127412Z 2022-11-23T02:32:19.6127643Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6127731Z Ran 1 test in 4.242s 2022-11-23T02:32:19.6127737Z 2022-11-23T02:32:19.6127805Z OK 2022-11-23T02:32:19.6127810Z 2022-11-23T02:32:19.6127905Z Generating XML reports... 2022-11-23T02:32:19.6128324Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021219.xml 2022-11-23T02:32:19.6128650Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6128984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6129131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6129475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6129633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6129640Z 2022-11-23T02:32:19.6129725Z Running tests... 2022-11-23T02:32:19.6129963Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6130255Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52590 2022-11-23T02:32:19.6130446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52591 2022-11-23T02:32:19.6130675Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6131042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6131193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6131542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6131699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6131905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6132271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6132603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6132745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6133086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6133242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6133449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6133808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6134002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6134186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6134500Z STAGE:2022-11-23 02:12:29 52591:52591 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6134794Z STAGE:2022-11-23 02:12:29 52590:52590 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6135100Z STAGE:2022-11-23 02:12:29 52591:52591 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6135404Z STAGE:2022-11-23 02:12:29 52590:52590 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6135732Z STAGE:2022-11-23 02:12:29 52591:52591 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6136049Z STAGE:2022-11-23 02:12:29 52590:52590 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6136252Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6136453Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6136794Z STAGE:2022-11-23 02:12:29 52590:52590 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6137093Z STAGE:2022-11-23 02:12:29 52591:52591 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6137402Z STAGE:2022-11-23 02:12:29 52590:52590 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6137716Z STAGE:2022-11-23 02:12:29 52590:52590 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6138020Z STAGE:2022-11-23 02:12:29 52591:52591 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6138335Z STAGE:2022-11-23 02:12:29 52591:52591 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6139047Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:32:19.6139138Z warnings.warn( 2022-11-23T02:32:19.6139865Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:32:19.6139952Z warnings.warn( 2022-11-23T02:32:19.6140026Z ok (4.336s) 2022-11-23T02:32:19.6140032Z 2022-11-23T02:32:19.6140265Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6140353Z Ran 1 test in 4.337s 2022-11-23T02:32:19.6140359Z 2022-11-23T02:32:19.6140428Z OK 2022-11-23T02:32:19.6140433Z 2022-11-23T02:32:19.6140529Z Generating XML reports... 2022-11-23T02:32:19.6140948Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021227.xml 2022-11-23T02:32:19.6141231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6141566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6141714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6142061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6142221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6142227Z 2022-11-23T02:32:19.6142309Z Running tests... 2022-11-23T02:32:19.6142549Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6142830Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52779 2022-11-23T02:32:19.6143021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52780 2022-11-23T02:32:19.6143254Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6143588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6143738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6144078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6144239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6144450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6144781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6144925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6145301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6145457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6145671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6146035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6146394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6146587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6146889Z STAGE:2022-11-23 02:12:36 52780:52780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6147079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6147388Z STAGE:2022-11-23 02:12:36 52779:52779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6147697Z STAGE:2022-11-23 02:12:36 52780:52780 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6148033Z STAGE:2022-11-23 02:12:36 52779:52779 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6148354Z STAGE:2022-11-23 02:12:36 52780:52780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6148672Z STAGE:2022-11-23 02:12:36 52779:52779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6148970Z STAGE:2022-11-23 02:12:36 52779:52779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6149264Z STAGE:2022-11-23 02:12:36 52780:52780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6149569Z STAGE:2022-11-23 02:12:36 52779:52779 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6149884Z STAGE:2022-11-23 02:12:36 52779:52779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6150184Z STAGE:2022-11-23 02:12:36 52780:52780 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6150501Z STAGE:2022-11-23 02:12:36 52780:52780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6150579Z ok (3.954s) 2022-11-23T02:32:19.6150586Z 2022-11-23T02:32:19.6150824Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6150909Z Ran 1 test in 3.954s 2022-11-23T02:32:19.6150916Z 2022-11-23T02:32:19.6150985Z OK 2022-11-23T02:32:19.6150990Z 2022-11-23T02:32:19.6151076Z Generating XML reports... 2022-11-23T02:32:19.6151490Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021234.xml 2022-11-23T02:32:19.6151775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6152111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6152255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6152602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6152761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6152767Z 2022-11-23T02:32:19.6152850Z Running tests... 2022-11-23T02:32:19.6153088Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6153372Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52968 2022-11-23T02:32:19.6153556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52969 2022-11-23T02:32:19.6153848Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6154184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6154332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6154676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6154834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6155040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6155402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6155730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6155877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6156288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6156492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6156697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6157049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6157246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6157436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6157514Z ok (4.245s) 2022-11-23T02:32:19.6157523Z 2022-11-23T02:32:19.6157765Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6157855Z Ran 1 test in 4.245s 2022-11-23T02:32:19.6157861Z 2022-11-23T02:32:19.6157928Z OK 2022-11-23T02:32:19.6157934Z 2022-11-23T02:32:19.6158032Z Generating XML reports... 2022-11-23T02:32:19.6158452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021241.xml 2022-11-23T02:32:19.6158733Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6159068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6159215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6159560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6159715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6159724Z 2022-11-23T02:32:19.6159806Z Running tests... 2022-11-23T02:32:19.6160045Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6160318Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53151 2022-11-23T02:32:19.6160505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53152 2022-11-23T02:32:19.6160740Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6161073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6161218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6161559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6161755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6161963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6162334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6162670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6162812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6163150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6163308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6163513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6163877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6164071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6164418Z STAGE:2022-11-23 02:12:51 53152:53152 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6164614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6164921Z STAGE:2022-11-23 02:12:51 53151:53151 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6165228Z STAGE:2022-11-23 02:12:51 53151:53151 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6165544Z STAGE:2022-11-23 02:12:51 53151:53151 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6165846Z STAGE:2022-11-23 02:12:51 53152:53152 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6166167Z STAGE:2022-11-23 02:12:51 53152:53152 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6166465Z STAGE:2022-11-23 02:12:51 53151:53151 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6166769Z STAGE:2022-11-23 02:12:51 53152:53152 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6167075Z STAGE:2022-11-23 02:12:51 53152:53152 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6167600Z STAGE:2022-11-23 02:12:51 53151:53151 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:12:51 53152:53152 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6167606Z 2022-11-23T02:32:19.6167918Z STAGE:2022-11-23 02:12:51 53151:53151 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6167995Z ok (3.941s) 2022-11-23T02:32:19.6168001Z 2022-11-23T02:32:19.6168241Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6168326Z Ran 1 test in 3.942s 2022-11-23T02:32:19.6168332Z 2022-11-23T02:32:19.6168390Z OK 2022-11-23T02:32:19.6168404Z 2022-11-23T02:32:19.6168490Z Generating XML reports... 2022-11-23T02:32:19.6168909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021248.xml 2022-11-23T02:32:19.6169194Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6169530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6169677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6170022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6170181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6170225Z 2022-11-23T02:32:19.6170308Z Running tests... 2022-11-23T02:32:19.6170545Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6170830Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53340 2022-11-23T02:32:19.6171019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53341 2022-11-23T02:32:19.6171252Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6171591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6171738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6172080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6172242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6172454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6172789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6172971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6173318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6173478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6173686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6174188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6174554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6174749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6175057Z STAGE:2022-11-23 02:12:58 53341:53341 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6175251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6175552Z STAGE:2022-11-23 02:12:58 53340:53340 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6176064Z STAGE:2022-11-23 02:12:58 53341:53341 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:12:58 53340:53340 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6176071Z 2022-11-23T02:32:19.6176608Z STAGE:2022-11-23 02:12:58 53341:53341 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:12:58 53340:53340 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6176617Z 2022-11-23T02:32:19.6176918Z STAGE:2022-11-23 02:12:58 53340:53340 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6177213Z STAGE:2022-11-23 02:12:58 53341:53341 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6177519Z STAGE:2022-11-23 02:12:58 53341:53341 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6177834Z STAGE:2022-11-23 02:12:58 53341:53341 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6178138Z STAGE:2022-11-23 02:12:58 53340:53340 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6178452Z STAGE:2022-11-23 02:12:58 53340:53340 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6178529Z ok (4.039s) 2022-11-23T02:32:19.6178535Z 2022-11-23T02:32:19.6178822Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6178910Z Ran 1 test in 4.039s 2022-11-23T02:32:19.6178916Z 2022-11-23T02:32:19.6178983Z OK 2022-11-23T02:32:19.6178988Z 2022-11-23T02:32:19.6179082Z Generating XML reports... 2022-11-23T02:32:19.6179500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021255.xml 2022-11-23T02:32:19.6179786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6180123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6180269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6180615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6180772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6180780Z 2022-11-23T02:32:19.6180860Z Running tests... 2022-11-23T02:32:19.6181092Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6181371Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53529 2022-11-23T02:32:19.6181596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53530 2022-11-23T02:32:19.6181831Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6182167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6182314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6182658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6182815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6183026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6183352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6183499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6183841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6184069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6184277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6184641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6184997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6185195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6185380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6185888Z STAGE:2022-11-23 02:13:05 53530:53530 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 02:13:05 53529:53529 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6185894Z 2022-11-23T02:32:19.6186202Z STAGE:2022-11-23 02:13:05 53530:53530 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6186524Z STAGE:2022-11-23 02:13:05 53530:53530 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6186833Z STAGE:2022-11-23 02:13:05 53529:53529 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6187193Z STAGE:2022-11-23 02:13:05 53529:53529 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6187498Z STAGE:2022-11-23 02:13:05 53530:53530 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6187790Z STAGE:2022-11-23 02:13:05 53529:53529 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6188095Z STAGE:2022-11-23 02:13:05 53529:53529 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6188400Z STAGE:2022-11-23 02:13:05 53529:53529 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6188701Z STAGE:2022-11-23 02:13:05 53530:53530 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6189013Z STAGE:2022-11-23 02:13:05 53530:53530 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6189090Z ok (3.851s) 2022-11-23T02:32:19.6189095Z 2022-11-23T02:32:19.6189335Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6189425Z Ran 1 test in 3.851s 2022-11-23T02:32:19.6189431Z 2022-11-23T02:32:19.6189499Z OK 2022-11-23T02:32:19.6189504Z 2022-11-23T02:32:19.6189601Z Generating XML reports... 2022-11-23T02:32:19.6190055Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021302.xml 2022-11-23T02:32:19.6190337Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6190673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6190816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6191160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6191318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6191327Z 2022-11-23T02:32:19.6191411Z Running tests... 2022-11-23T02:32:19.6191654Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6191932Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53718 2022-11-23T02:32:19.6192120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53719 2022-11-23T02:32:19.6192353Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6192690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6192837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6193181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6193342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6193541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6193902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6194236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6194379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6194722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6194880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6195087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6195444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6195680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6195866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6196258Z STAGE:2022-11-23 02:13:12 53719:53719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6196559Z STAGE:2022-11-23 02:13:12 53718:53718 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6196864Z STAGE:2022-11-23 02:13:12 53718:53718 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6197286Z STAGE:2022-11-23 02:13:12 53718:53718 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6197469Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6197738Z STAGE:2022-11-23 02:13:12 53719:53719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6198018Z STAGE:2022-11-23 02:13:12 53719:53719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6198201Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6198508Z STAGE:2022-11-23 02:13:12 53718:53718 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6198776Z STAGE:2022-11-23 02:13:12 53719:53719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6199045Z STAGE:2022-11-23 02:13:12 53719:53719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6199322Z STAGE:2022-11-23 02:13:12 53719:53719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6199595Z STAGE:2022-11-23 02:13:12 53718:53718 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6199866Z STAGE:2022-11-23 02:13:12 53718:53718 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6200133Z STAGE:2022-11-23 02:13:12 53719:53719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6200397Z STAGE:2022-11-23 02:13:12 53718:53718 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6200671Z STAGE:2022-11-23 02:13:12 53718:53718 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6201135Z STAGE:2022-11-23 02:13:12 53719:53719 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:13:12 53718:53718 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6201140Z 2022-11-23T02:32:19.6201420Z STAGE:2022-11-23 02:13:12 53719:53719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6201490Z ok (4.440s) 2022-11-23T02:32:19.6201495Z 2022-11-23T02:32:19.6201709Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6201786Z Ran 1 test in 4.441s 2022-11-23T02:32:19.6201791Z 2022-11-23T02:32:19.6201852Z OK 2022-11-23T02:32:19.6201857Z 2022-11-23T02:32:19.6201946Z Generating XML reports... 2022-11-23T02:32:19.6202317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021309.xml 2022-11-23T02:32:19.6202569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6202871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6202999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6203305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6203447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6203494Z 2022-11-23T02:32:19.6203570Z Running tests... 2022-11-23T02:32:19.6203783Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6204035Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53907 2022-11-23T02:32:19.6204205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53908 2022-11-23T02:32:19.6204410Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6204705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6204833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6205138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6205272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6205459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6205753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6205946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6206257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6206398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6206583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6207027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6207336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6207505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6207669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6207933Z STAGE:2022-11-23 02:13:19 53908:53908 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6208187Z STAGE:2022-11-23 02:13:19 53907:53907 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6208444Z STAGE:2022-11-23 02:13:20 53907:53907 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6208711Z STAGE:2022-11-23 02:13:20 53907:53907 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6208887Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6209143Z STAGE:2022-11-23 02:13:20 53908:53908 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6209414Z STAGE:2022-11-23 02:13:20 53908:53908 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6209580Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6209833Z STAGE:2022-11-23 02:13:20 53907:53907 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6210078Z STAGE:2022-11-23 02:13:20 53908:53908 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6210334Z STAGE:2022-11-23 02:13:20 53908:53908 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6210595Z STAGE:2022-11-23 02:13:20 53908:53908 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6210842Z STAGE:2022-11-23 02:13:20 53907:53907 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6211144Z STAGE:2022-11-23 02:13:20 53907:53907 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6211397Z STAGE:2022-11-23 02:13:20 53908:53908 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6211646Z STAGE:2022-11-23 02:13:20 53907:53907 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6211900Z STAGE:2022-11-23 02:13:20 53907:53907 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6212340Z STAGE:2022-11-23 02:13:20 53907:53907 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:13:20 53908:53908 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6212345Z 2022-11-23T02:32:19.6212608Z STAGE:2022-11-23 02:13:20 53908:53908 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6212672Z ok (4.542s) 2022-11-23T02:32:19.6212677Z 2022-11-23T02:32:19.6212879Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6212953Z Ran 1 test in 4.542s 2022-11-23T02:32:19.6212957Z 2022-11-23T02:32:19.6213013Z OK 2022-11-23T02:32:19.6213017Z 2022-11-23T02:32:19.6213097Z Generating XML reports... 2022-11-23T02:32:19.6213496Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021317.xml 2022-11-23T02:32:19.6213738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6214020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6214142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6214434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6214569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6214577Z 2022-11-23T02:32:19.6214647Z Running tests... 2022-11-23T02:32:19.6214849Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6215091Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54096 2022-11-23T02:32:19.6215250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54097 2022-11-23T02:32:19.6215447Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6215727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6215848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6216126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6216266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6216441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6216720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6216845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6217136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6217268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6217445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6217751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6218057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6218257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6218420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6218683Z STAGE:2022-11-23 02:13:27 54097:54097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6218937Z STAGE:2022-11-23 02:13:27 54096:54096 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6219193Z STAGE:2022-11-23 02:13:27 54096:54096 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6219458Z STAGE:2022-11-23 02:13:27 54096:54096 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6219715Z STAGE:2022-11-23 02:13:27 54097:54097 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6219890Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6220155Z STAGE:2022-11-23 02:13:27 54097:54097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6220320Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:32:19.6220609Z STAGE:2022-11-23 02:13:27 54096:54096 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6220861Z STAGE:2022-11-23 02:13:27 54097:54097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6221116Z STAGE:2022-11-23 02:13:27 54096:54096 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6221361Z STAGE:2022-11-23 02:13:27 54097:54097 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6221625Z STAGE:2022-11-23 02:13:27 54096:54096 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6221894Z STAGE:2022-11-23 02:13:27 54097:54097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6222146Z STAGE:2022-11-23 02:13:27 54096:54096 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6222395Z STAGE:2022-11-23 02:13:27 54097:54097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6222651Z STAGE:2022-11-23 02:13:27 54097:54097 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6222917Z STAGE:2022-11-23 02:13:27 54097:54097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6223171Z STAGE:2022-11-23 02:13:27 54096:54096 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6223434Z STAGE:2022-11-23 02:13:27 54096:54096 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6223500Z ok (4.442s) 2022-11-23T02:32:19.6223506Z 2022-11-23T02:32:19.6223713Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6223788Z Ran 1 test in 4.442s 2022-11-23T02:32:19.6223793Z 2022-11-23T02:32:19.6223853Z OK 2022-11-23T02:32:19.6223857Z 2022-11-23T02:32:19.6223938Z Generating XML reports... 2022-11-23T02:32:19.6224286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021324.xml 2022-11-23T02:32:19.6224526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6224809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6224932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6225216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6225346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6225384Z 2022-11-23T02:32:19.6225453Z Running tests... 2022-11-23T02:32:19.6225656Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6225824Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:32:19.6225839Z 2022-11-23T02:32:19.6226029Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6226103Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6226107Z 2022-11-23T02:32:19.6226174Z OK (skipped=1) 2022-11-23T02:32:19.6226179Z 2022-11-23T02:32:19.6226256Z Generating XML reports... 2022-11-23T02:32:19.6226604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021332.xml 2022-11-23T02:32:19.6226842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6227122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6227250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6227544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6227711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6227716Z 2022-11-23T02:32:19.6227788Z Running tests... 2022-11-23T02:32:19.6227991Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6228178Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:32:19.6228183Z 2022-11-23T02:32:19.6228380Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6228452Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6228457Z 2022-11-23T02:32:19.6228525Z OK (skipped=1) 2022-11-23T02:32:19.6228529Z 2022-11-23T02:32:19.6228615Z Generating XML reports... 2022-11-23T02:32:19.6228960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021335.xml 2022-11-23T02:32:19.6229196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6229479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6229602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6229888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6230010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6230022Z 2022-11-23T02:32:19.6230081Z Running tests... 2022-11-23T02:32:19.6230282Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6230470Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s) 2022-11-23T02:32:19.6230477Z 2022-11-23T02:32:19.6230672Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6230741Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6230745Z 2022-11-23T02:32:19.6230813Z OK (skipped=1) 2022-11-23T02:32:19.6230818Z 2022-11-23T02:32:19.6230901Z Generating XML reports... 2022-11-23T02:32:19.6231242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021338.xml 2022-11-23T02:32:19.6231479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6231756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6231877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6232164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6232328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6232334Z 2022-11-23T02:32:19.6232403Z Running tests... 2022-11-23T02:32:19.6232606Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6232805Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s) 2022-11-23T02:32:19.6232811Z 2022-11-23T02:32:19.6233011Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6233084Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6233089Z 2022-11-23T02:32:19.6233156Z OK (skipped=1) 2022-11-23T02:32:19.6233161Z 2022-11-23T02:32:19.6233240Z Generating XML reports... 2022-11-23T02:32:19.6233583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021341.xml 2022-11-23T02:32:19.6233819Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6234092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6234213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6234535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6234668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6234673Z 2022-11-23T02:32:19.6234742Z Running tests... 2022-11-23T02:32:19.6234939Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6235124Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:32:19.6235129Z 2022-11-23T02:32:19.6235323Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6235397Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6235401Z 2022-11-23T02:32:19.6235467Z OK (skipped=1) 2022-11-23T02:32:19.6235472Z 2022-11-23T02:32:19.6235553Z Generating XML reports... 2022-11-23T02:32:19.6235899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021344.xml 2022-11-23T02:32:19.6236201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6236480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6236601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6236888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6237020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6237025Z 2022-11-23T02:32:19.6237093Z Running tests... 2022-11-23T02:32:19.6237293Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6237487Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s) 2022-11-23T02:32:19.6237492Z 2022-11-23T02:32:19.6237694Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6237763Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6237768Z 2022-11-23T02:32:19.6237826Z OK (skipped=1) 2022-11-23T02:32:19.6237838Z 2022-11-23T02:32:19.6237909Z Generating XML reports... 2022-11-23T02:32:19.6238254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021346.xml 2022-11-23T02:32:19.6238494Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6238772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6238936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6239227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6239362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6239368Z 2022-11-23T02:32:19.6239440Z Running tests... 2022-11-23T02:32:19.6239639Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6239825Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:32:19.6239830Z 2022-11-23T02:32:19.6240028Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6240101Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6240105Z 2022-11-23T02:32:19.6240174Z OK (skipped=1) 2022-11-23T02:32:19.6240178Z 2022-11-23T02:32:19.6240259Z Generating XML reports... 2022-11-23T02:32:19.6240604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021349.xml 2022-11-23T02:32:19.6240841Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6241155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6241278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6241568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6241698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6241703Z 2022-11-23T02:32:19.6241772Z Running tests... 2022-11-23T02:32:19.6241971Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6242161Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:32:19.6242176Z 2022-11-23T02:32:19.6242363Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6242436Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6242440Z 2022-11-23T02:32:19.6242507Z OK (skipped=1) 2022-11-23T02:32:19.6242511Z 2022-11-23T02:32:19.6242592Z Generating XML reports... 2022-11-23T02:32:19.6242935Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021352.xml 2022-11-23T02:32:19.6243171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6243452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6243575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6243865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6244000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6244004Z 2022-11-23T02:32:19.6244072Z Running tests... 2022-11-23T02:32:19.6244269Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6244478Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:32:19.6244484Z 2022-11-23T02:32:19.6244678Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6244751Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6244756Z 2022-11-23T02:32:19.6244824Z OK (skipped=1) 2022-11-23T02:32:19.6244828Z 2022-11-23T02:32:19.6244910Z Generating XML reports... 2022-11-23T02:32:19.6245246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021355.xml 2022-11-23T02:32:19.6245480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6245798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6245919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6246206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6246329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6246340Z 2022-11-23T02:32:19.6246399Z Running tests... 2022-11-23T02:32:19.6246600Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6246812Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:32:19.6246817Z 2022-11-23T02:32:19.6247013Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6247088Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6247093Z 2022-11-23T02:32:19.6247158Z OK (skipped=1) 2022-11-23T02:32:19.6247162Z 2022-11-23T02:32:19.6247243Z Generating XML reports... 2022-11-23T02:32:19.6247584Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021358.xml 2022-11-23T02:32:19.6247849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6248133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6248253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6248541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6248675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6248680Z 2022-11-23T02:32:19.6248751Z Running tests... 2022-11-23T02:32:19.6248955Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6249166Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:32:19.6249172Z 2022-11-23T02:32:19.6249371Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6249445Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6249450Z 2022-11-23T02:32:19.6249516Z OK (skipped=1) 2022-11-23T02:32:19.6249520Z 2022-11-23T02:32:19.6249602Z Generating XML reports... 2022-11-23T02:32:19.6249945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021401.xml 2022-11-23T02:32:19.6250181Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6250455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6250575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6250864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6250996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6251003Z 2022-11-23T02:32:19.6251076Z Running tests... 2022-11-23T02:32:19.6251273Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6251494Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s) 2022-11-23T02:32:19.6251499Z 2022-11-23T02:32:19.6251693Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6251767Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6251772Z 2022-11-23T02:32:19.6251839Z OK (skipped=1) 2022-11-23T02:32:19.6251843Z 2022-11-23T02:32:19.6251927Z Generating XML reports... 2022-11-23T02:32:19.6252301Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021404.xml 2022-11-23T02:32:19.6252542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6252824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6252946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6253238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6253373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6253377Z 2022-11-23T02:32:19.6253447Z Running tests... 2022-11-23T02:32:19.6253651Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6253866Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:32:19.6253874Z 2022-11-23T02:32:19.6254068Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6254140Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6254145Z 2022-11-23T02:32:19.6254211Z OK (skipped=1) 2022-11-23T02:32:19.6254248Z 2022-11-23T02:32:19.6254329Z Generating XML reports... 2022-11-23T02:32:19.6254664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021407.xml 2022-11-23T02:32:19.6254904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6255184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6255308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6255593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6255727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6255732Z 2022-11-23T02:32:19.6255803Z Running tests... 2022-11-23T02:32:19.6255999Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6256225Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s) 2022-11-23T02:32:19.6256230Z 2022-11-23T02:32:19.6256426Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6256496Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6256501Z 2022-11-23T02:32:19.6256572Z OK (skipped=1) 2022-11-23T02:32:19.6256576Z 2022-11-23T02:32:19.6256655Z Generating XML reports... 2022-11-23T02:32:19.6256998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021410.xml 2022-11-23T02:32:19.6257236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6257518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6257640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6257929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6258062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6258067Z 2022-11-23T02:32:19.6258136Z Running tests... 2022-11-23T02:32:19.6258338Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6258550Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:32:19.6258556Z 2022-11-23T02:32:19.6258749Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6258843Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6258859Z 2022-11-23T02:32:19.6258919Z OK (skipped=1) 2022-11-23T02:32:19.6258923Z 2022-11-23T02:32:19.6259002Z Generating XML reports... 2022-11-23T02:32:19.6259346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021413.xml 2022-11-23T02:32:19.6259580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6259859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6259978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6260264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6260393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6260400Z 2022-11-23T02:32:19.6260470Z Running tests... 2022-11-23T02:32:19.6260669Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6260889Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s) 2022-11-23T02:32:19.6260926Z 2022-11-23T02:32:19.6261127Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6261201Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6261206Z 2022-11-23T02:32:19.6261276Z OK (skipped=1) 2022-11-23T02:32:19.6261281Z 2022-11-23T02:32:19.6261362Z Generating XML reports... 2022-11-23T02:32:19.6261704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021416.xml 2022-11-23T02:32:19.6261941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6262222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6262346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6262633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6262768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6262772Z 2022-11-23T02:32:19.6262841Z Running tests... 2022-11-23T02:32:19.6263031Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6263238Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:32:19.6263243Z 2022-11-23T02:32:19.6263438Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6263510Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6263515Z 2022-11-23T02:32:19.6263586Z OK (skipped=1) 2022-11-23T02:32:19.6263591Z 2022-11-23T02:32:19.6263672Z Generating XML reports... 2022-11-23T02:32:19.6264010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021419.xml 2022-11-23T02:32:19.6264247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6264527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6264648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6264939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6265075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6265080Z 2022-11-23T02:32:19.6265151Z Running tests... 2022-11-23T02:32:19.6265349Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6265624Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:32:19.6265629Z 2022-11-23T02:32:19.6265828Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6265903Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6265910Z 2022-11-23T02:32:19.6265980Z OK (skipped=1) 2022-11-23T02:32:19.6265985Z 2022-11-23T02:32:19.6266067Z Generating XML reports... 2022-11-23T02:32:19.6266409Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021422.xml 2022-11-23T02:32:19.6266649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6266934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6267058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6267338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6267473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6267478Z 2022-11-23T02:32:19.6267552Z Running tests... 2022-11-23T02:32:19.6267786Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6268002Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:32:19.6268007Z 2022-11-23T02:32:19.6268208Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6268280Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6268285Z 2022-11-23T02:32:19.6268355Z OK (skipped=1) 2022-11-23T02:32:19.6268359Z 2022-11-23T02:32:19.6268440Z Generating XML reports... 2022-11-23T02:32:19.6268785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021425.xml 2022-11-23T02:32:19.6269023Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6269302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6269426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6269715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6269848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6269853Z 2022-11-23T02:32:19.6269924Z Running tests... 2022-11-23T02:32:19.6270272Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6270498Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:32:19.6270506Z 2022-11-23T02:32:19.6270701Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6270773Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6270778Z 2022-11-23T02:32:19.6270845Z OK (skipped=1) 2022-11-23T02:32:19.6270850Z 2022-11-23T02:32:19.6270931Z Generating XML reports... 2022-11-23T02:32:19.6271274Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021427.xml 2022-11-23T02:32:19.6271501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6271783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6271904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6272193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6272358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6272363Z 2022-11-23T02:32:19.6272431Z Running tests... 2022-11-23T02:32:19.6272630Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6272848Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:32:19.6272853Z 2022-11-23T02:32:19.6273047Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6273119Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6273124Z 2022-11-23T02:32:19.6273191Z OK (skipped=1) 2022-11-23T02:32:19.6273195Z 2022-11-23T02:32:19.6273276Z Generating XML reports... 2022-11-23T02:32:19.6273617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021430.xml 2022-11-23T02:32:19.6273855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6274138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6274257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6274579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6274715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6274720Z 2022-11-23T02:32:19.6274789Z Running tests... 2022-11-23T02:32:19.6274993Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6275224Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:32:19.6275229Z 2022-11-23T02:32:19.6275424Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6275498Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6275502Z 2022-11-23T02:32:19.6275570Z OK (skipped=1) 2022-11-23T02:32:19.6275575Z 2022-11-23T02:32:19.6275646Z Generating XML reports... 2022-11-23T02:32:19.6275988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021433.xml 2022-11-23T02:32:19.6276282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6276566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6276690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6276978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6277111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6277116Z 2022-11-23T02:32:19.6277184Z Running tests... 2022-11-23T02:32:19.6277387Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6277600Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:32:19.6277605Z 2022-11-23T02:32:19.6277807Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6277881Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6277886Z 2022-11-23T02:32:19.6277955Z OK (skipped=1) 2022-11-23T02:32:19.6277960Z 2022-11-23T02:32:19.6278043Z Generating XML reports... 2022-11-23T02:32:19.6278390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021436.xml 2022-11-23T02:32:19.6278632Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6278913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6279077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6279370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6279503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6279511Z 2022-11-23T02:32:19.6279581Z Running tests... 2022-11-23T02:32:19.6279780Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6279994Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s) 2022-11-23T02:32:19.6280009Z 2022-11-23T02:32:19.6280194Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6280265Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6280269Z 2022-11-23T02:32:19.6280337Z OK (skipped=1) 2022-11-23T02:32:19.6280341Z 2022-11-23T02:32:19.6280421Z Generating XML reports... 2022-11-23T02:32:19.6280764Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021439.xml 2022-11-23T02:32:19.6281000Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6281314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6281438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6281725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6281854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6281859Z 2022-11-23T02:32:19.6281927Z Running tests... 2022-11-23T02:32:19.6282124Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6282359Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55677 2022-11-23T02:32:19.6282519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55678 2022-11-23T02:32:19.6282713Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6282995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6283119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6283407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6283539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6283713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6301862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6302047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6302409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6302551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6302736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6303056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6303366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6303535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6303858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6304036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6304214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6304525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6304828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6304899Z ok (4.842s) 2022-11-23T02:32:19.6304906Z 2022-11-23T02:32:19.6305102Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6305180Z Ran 1 test in 4.842s 2022-11-23T02:32:19.6305185Z 2022-11-23T02:32:19.6305245Z OK 2022-11-23T02:32:19.6305249Z 2022-11-23T02:32:19.6305335Z Generating XML reports... 2022-11-23T02:32:19.6305692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021442.xml 2022-11-23T02:32:19.6305932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6306260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6306387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6306685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6306822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6306827Z 2022-11-23T02:32:19.6306903Z Running tests... 2022-11-23T02:32:19.6307109Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6307442Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s) 2022-11-23T02:32:19.6307450Z 2022-11-23T02:32:19.6307648Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6307726Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6307731Z 2022-11-23T02:32:19.6307804Z OK (skipped=1) 2022-11-23T02:32:19.6307811Z 2022-11-23T02:32:19.6307897Z Generating XML reports... 2022-11-23T02:32:19.6308243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021450.xml 2022-11-23T02:32:19.6308488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6308767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6308889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6309180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6309316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6309320Z 2022-11-23T02:32:19.6309381Z Running tests... 2022-11-23T02:32:19.6309575Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6309891Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s) 2022-11-23T02:32:19.6309896Z 2022-11-23T02:32:19.6310097Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6310172Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6310177Z 2022-11-23T02:32:19.6310249Z OK (skipped=1) 2022-11-23T02:32:19.6310254Z 2022-11-23T02:32:19.6310339Z Generating XML reports... 2022-11-23T02:32:19.6310689Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021453.xml 2022-11-23T02:32:19.6310964Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6311242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6311360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6311648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6311776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6311781Z 2022-11-23T02:32:19.6311846Z Running tests... 2022-11-23T02:32:19.6312040Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6312258Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55989 2022-11-23T02:32:19.6312411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55990 2022-11-23T02:32:19.6312604Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6312883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6313034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6313321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6313451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6313623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6313899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6314016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6314301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6314427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6314601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6314905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6315209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6315369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6315521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6315582Z ok (4.639s) 2022-11-23T02:32:19.6315587Z 2022-11-23T02:32:19.6315787Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6315856Z Ran 1 test in 4.640s 2022-11-23T02:32:19.6315861Z 2022-11-23T02:32:19.6315913Z OK 2022-11-23T02:32:19.6315919Z 2022-11-23T02:32:19.6315998Z Generating XML reports... 2022-11-23T02:32:19.6316435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021456.xml 2022-11-23T02:32:19.6316673Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6316951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6317071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6317356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6317487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6317537Z 2022-11-23T02:32:19.6317605Z Running tests... 2022-11-23T02:32:19.6317802Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6318030Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56172 2022-11-23T02:32:19.6318185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56173 2022-11-23T02:32:19.6318378Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6318656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6318775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6319061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6319191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6319365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6319661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6319974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6320094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6320379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6320508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6320679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6320983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6321146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6321304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6321367Z ok (5.043s) 2022-11-23T02:32:19.6321372Z 2022-11-23T02:32:19.6321569Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6321640Z Ran 1 test in 5.044s 2022-11-23T02:32:19.6321645Z 2022-11-23T02:32:19.6321699Z OK 2022-11-23T02:32:19.6321703Z 2022-11-23T02:32:19.6321780Z Generating XML reports... 2022-11-23T02:32:19.6322123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021503.xml 2022-11-23T02:32:19.6322356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6322634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6322750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6323034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6323163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6323169Z 2022-11-23T02:32:19.6323234Z Running tests... 2022-11-23T02:32:19.6323428Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6323650Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56355 2022-11-23T02:32:19.6323802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56356 2022-11-23T02:32:19.6323994Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6324308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6324426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6324712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6324842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6325014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6325288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6325406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6325687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6325817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6325989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6326323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6326626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6326786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6326943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6327112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6327279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6327579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6327877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6327941Z ok (4.558s) 2022-11-23T02:32:19.6327946Z 2022-11-23T02:32:19.6328146Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6328210Z Ran 1 test in 4.559s 2022-11-23T02:32:19.6328220Z 2022-11-23T02:32:19.6328269Z OK 2022-11-23T02:32:19.6328273Z 2022-11-23T02:32:19.6328353Z Generating XML reports... 2022-11-23T02:32:19.6328697Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021511.xml 2022-11-23T02:32:19.6328931Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6329207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6329329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6329613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6329746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6329752Z 2022-11-23T02:32:19.6329817Z Running tests... 2022-11-23T02:32:19.6330016Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6330258Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56544 2022-11-23T02:32:19.6330418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56545 2022-11-23T02:32:19.6330617Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6330936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6331060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6331351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6331486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6331663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6331949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6332075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6332367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6332506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6332670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6333034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6333340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6333499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6333660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6333767Z skip: Skipped due to small world size. (3.842s) 2022-11-23T02:32:19.6333772Z 2022-11-23T02:32:19.6333970Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6334041Z Ran 1 test in 3.843s 2022-11-23T02:32:19.6334047Z 2022-11-23T02:32:19.6334111Z OK (skipped=1) 2022-11-23T02:32:19.6334116Z 2022-11-23T02:32:19.6334193Z Generating XML reports... 2022-11-23T02:32:19.6334536Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021519.xml 2022-11-23T02:32:19.6334772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6335047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6335168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6335457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6335585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6335590Z 2022-11-23T02:32:19.6335656Z Running tests... 2022-11-23T02:32:19.6335855Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6336079Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56727 2022-11-23T02:32:19.6336234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56728 2022-11-23T02:32:19.6336424Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6336699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6336819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6337099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6337229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6337437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6337740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6338018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6338135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6338421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6338552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6338722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6339022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6339182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6339341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6339448Z skip: Skipped due to small world size. (3.752s) 2022-11-23T02:32:19.6339453Z 2022-11-23T02:32:19.6339683Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6339753Z Ran 1 test in 3.752s 2022-11-23T02:32:19.6339758Z 2022-11-23T02:32:19.6339822Z OK (skipped=1) 2022-11-23T02:32:19.6339826Z 2022-11-23T02:32:19.6339905Z Generating XML reports... 2022-11-23T02:32:19.6340247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021526.xml 2022-11-23T02:32:19.6340480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6340755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6340875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6341156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6341283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6341294Z 2022-11-23T02:32:19.6341355Z Running tests... 2022-11-23T02:32:19.6341553Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6341783Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56910 2022-11-23T02:32:19.6341935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56911 2022-11-23T02:32:19.6342124Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6342403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6342520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6342806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6342936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6343105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6343379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6343496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6343777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6343937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6344106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6344405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6344705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6344863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6345020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6345124Z skip: Skipped due to small world size. (3.541s) 2022-11-23T02:32:19.6345129Z 2022-11-23T02:32:19.6345326Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6345394Z Ran 1 test in 3.542s 2022-11-23T02:32:19.6345401Z 2022-11-23T02:32:19.6345460Z OK (skipped=1) 2022-11-23T02:32:19.6345470Z 2022-11-23T02:32:19.6345543Z Generating XML reports... 2022-11-23T02:32:19.6345881Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021533.xml 2022-11-23T02:32:19.6346145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6346425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6346542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6346826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6346954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6346959Z 2022-11-23T02:32:19.6347024Z Running tests... 2022-11-23T02:32:19.6347223Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6347461Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57093 2022-11-23T02:32:19.6347613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57094 2022-11-23T02:32:19.6347803Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6348080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6348198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6348481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6348610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6348785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6349061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6349179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6349465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6349592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6349757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6350061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6350362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6350556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6350713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6350879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6351048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6351347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6351640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6351701Z ok (5.143s) 2022-11-23T02:32:19.6351706Z 2022-11-23T02:32:19.6351902Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6351970Z Ran 1 test in 5.144s 2022-11-23T02:32:19.6351977Z 2022-11-23T02:32:19.6352030Z OK 2022-11-23T02:32:19.6352035Z 2022-11-23T02:32:19.6352110Z Generating XML reports... 2022-11-23T02:32:19.6352451Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021539.xml 2022-11-23T02:32:19.6352717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6352998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6353115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6353398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6353527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6353532Z 2022-11-23T02:32:19.6353597Z Running tests... 2022-11-23T02:32:19.6353795Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6354027Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57282 2022-11-23T02:32:19.6354179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57283 2022-11-23T02:32:19.6354365Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6354638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6354755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6355040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6355168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6355341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6355616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6355732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6356017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6356198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6356368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6356669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6356968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6357188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6357345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6357512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6357682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6357984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6358283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6358346Z ok (4.945s) 2022-11-23T02:32:19.6358351Z 2022-11-23T02:32:19.6358547Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6358615Z Ran 1 test in 4.945s 2022-11-23T02:32:19.6358621Z 2022-11-23T02:32:19.6358673Z OK 2022-11-23T02:32:19.6358677Z 2022-11-23T02:32:19.6358749Z Generating XML reports... 2022-11-23T02:32:19.6359092Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021548.xml 2022-11-23T02:32:19.6359361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6359639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6359757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6360042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6360173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6360177Z 2022-11-23T02:32:19.6360246Z Running tests... 2022-11-23T02:32:19.6360447Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6360682Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 427 2022-11-23T02:32:19.6360833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 428 2022-11-23T02:32:19.6361029Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6361303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6361426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6361716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6361850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6362025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6362308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6362431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6362723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6362856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6363030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6363327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6363635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6363836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6363998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6364108Z skip: Skipped due to small world size. (3.741s) 2022-11-23T02:32:19.6364113Z 2022-11-23T02:32:19.6364320Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6364395Z Ran 1 test in 3.742s 2022-11-23T02:32:19.6364401Z 2022-11-23T02:32:19.6364472Z OK (skipped=1) 2022-11-23T02:32:19.6364477Z 2022-11-23T02:32:19.6364560Z Generating XML reports... 2022-11-23T02:32:19.6364908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021556.xml 2022-11-23T02:32:19.6365147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6365431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6365559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6365848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6366016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6366021Z 2022-11-23T02:32:19.6366093Z Running tests... 2022-11-23T02:32:19.6366296Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6366528Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 610 2022-11-23T02:32:19.6366683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 611 2022-11-23T02:32:19.6366877Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6367159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6367282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6367571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6367698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6367875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6368178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6368459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6368581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6368870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6369007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6369180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6369493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6369659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6369821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6369888Z ok (3.760s) 2022-11-23T02:32:19.6369893Z 2022-11-23T02:32:19.6370095Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6370169Z Ran 1 test in 3.760s 2022-11-23T02:32:19.6370173Z 2022-11-23T02:32:19.6370231Z OK 2022-11-23T02:32:19.6370269Z 2022-11-23T02:32:19.6370352Z Generating XML reports... 2022-11-23T02:32:19.6370700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021602.xml 2022-11-23T02:32:19.6370937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6371223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6371346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6371636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6371770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6371775Z 2022-11-23T02:32:19.6371835Z Running tests... 2022-11-23T02:32:19.6372038Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6372283Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 794 2022-11-23T02:32:19.6372437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 795 2022-11-23T02:32:19.6372813Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6373098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6373219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6373510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6373644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6373821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6374131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6374413Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6374535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6374825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6374956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6375131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6375441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6375609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6375778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6375846Z ok (3.739s) 2022-11-23T02:32:19.6375851Z 2022-11-23T02:32:19.6376056Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6376136Z Ran 1 test in 3.739s 2022-11-23T02:32:19.6376141Z 2022-11-23T02:32:19.6376190Z OK 2022-11-23T02:32:19.6376208Z 2022-11-23T02:32:19.6376279Z Generating XML reports... 2022-11-23T02:32:19.6376632Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021609.xml 2022-11-23T02:32:19.6376871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6377154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6377280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6377608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6377743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6377748Z 2022-11-23T02:32:19.6377823Z Running tests... 2022-11-23T02:32:19.6378029Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6378231Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:32:19.6378236Z 2022-11-23T02:32:19.6378438Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6378514Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6378519Z 2022-11-23T02:32:19.6378594Z OK (skipped=1) 2022-11-23T02:32:19.6378598Z 2022-11-23T02:32:19.6378692Z Generating XML reports... 2022-11-23T02:32:19.6379037Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021616.xml 2022-11-23T02:32:19.6379285Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6379576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6379734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6380028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6380162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6380167Z 2022-11-23T02:32:19.6380239Z Running tests... 2022-11-23T02:32:19.6380441Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6380619Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:32:19.6380634Z 2022-11-23T02:32:19.6380824Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6380897Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6380902Z 2022-11-23T02:32:19.6380972Z OK (skipped=1) 2022-11-23T02:32:19.6380976Z 2022-11-23T02:32:19.6381060Z Generating XML reports... 2022-11-23T02:32:19.6381404Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021619.xml 2022-11-23T02:32:19.6381646Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6381929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6382053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6382345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6382480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6382488Z 2022-11-23T02:32:19.6382559Z Running tests... 2022-11-23T02:32:19.6382758Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6382958Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:32:19.6382965Z 2022-11-23T02:32:19.6383164Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6383236Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6383241Z 2022-11-23T02:32:19.6383309Z OK (skipped=1) 2022-11-23T02:32:19.6383314Z 2022-11-23T02:32:19.6383403Z Generating XML reports... 2022-11-23T02:32:19.6383758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021622.xml 2022-11-23T02:32:19.6384000Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6384294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6384471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6384764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6384891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6384916Z 2022-11-23T02:32:19.6384976Z Running tests... 2022-11-23T02:32:19.6385185Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6385385Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:32:19.6385390Z 2022-11-23T02:32:19.6385586Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6385663Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6385668Z 2022-11-23T02:32:19.6385736Z OK (skipped=1) 2022-11-23T02:32:19.6385742Z 2022-11-23T02:32:19.6385822Z Generating XML reports... 2022-11-23T02:32:19.6386161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021625.xml 2022-11-23T02:32:19.6386395Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6386708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6386833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6387127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6387262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6387268Z 2022-11-23T02:32:19.6387337Z Running tests... 2022-11-23T02:32:19.6387535Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6387728Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:32:19.6387734Z 2022-11-23T02:32:19.6387929Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6387998Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6388003Z 2022-11-23T02:32:19.6388071Z OK (skipped=1) 2022-11-23T02:32:19.6388075Z 2022-11-23T02:32:19.6388154Z Generating XML reports... 2022-11-23T02:32:19.6388492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021628.xml 2022-11-23T02:32:19.6388720Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6388997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6389118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6389406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6389539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6389544Z 2022-11-23T02:32:19.6389612Z Running tests... 2022-11-23T02:32:19.6389811Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6390011Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:32:19.6390017Z 2022-11-23T02:32:19.6390214Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6390284Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6390289Z 2022-11-23T02:32:19.6390356Z OK (skipped=1) 2022-11-23T02:32:19.6390361Z 2022-11-23T02:32:19.6390439Z Generating XML reports... 2022-11-23T02:32:19.6390780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021631.xml 2022-11-23T02:32:19.6391049Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6391330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6391457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6391745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6391881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6391886Z 2022-11-23T02:32:19.6391956Z Running tests... 2022-11-23T02:32:19.6392155Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6392347Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:32:19.6392352Z 2022-11-23T02:32:19.6392550Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6392621Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6392626Z 2022-11-23T02:32:19.6392685Z OK (skipped=1) 2022-11-23T02:32:19.6392696Z 2022-11-23T02:32:19.6392767Z Generating XML reports... 2022-11-23T02:32:19.6393147Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021634.xml 2022-11-23T02:32:19.6393387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6393667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6393789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6394074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6394208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6394215Z 2022-11-23T02:32:19.6394284Z Running tests... 2022-11-23T02:32:19.6394481Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6394672Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:32:19.6394677Z 2022-11-23T02:32:19.6394874Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6394944Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6394949Z 2022-11-23T02:32:19.6395016Z OK (skipped=1) 2022-11-23T02:32:19.6395020Z 2022-11-23T02:32:19.6395100Z Generating XML reports... 2022-11-23T02:32:19.6395443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021637.xml 2022-11-23T02:32:19.6395678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6395958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6396133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6396429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6396565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6396570Z 2022-11-23T02:32:19.6396637Z Running tests... 2022-11-23T02:32:19.6396823Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6397046Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1443 2022-11-23T02:32:19.6397202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1444 2022-11-23T02:32:19.6397396Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6397719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6397842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6398131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6398267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6398441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6398722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6398842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6399131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6399262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6399438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6399745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6400086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6400251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6400510Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6400670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6400917Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6401176Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6401446Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6401698Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6401965Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6402205Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6402450Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6402701Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6402961Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6403214Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6403473Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6403719Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6403963Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6404212Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6404467Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6404708Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6404954Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6405245Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6405487Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6405735Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6405980Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6406235Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6406494Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6406739Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6406983Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6407231Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6407486Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6407765Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6408016Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6408261Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6408503Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6408753Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6409011Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6409260Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6409516Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6409762Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6410006Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6410255Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6410512Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6410759Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6411013Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6411259Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6411503Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6411752Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6412008Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6412255Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6412507Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6412752Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6412996Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6413279Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6413536Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6413777Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6414036Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6414277Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6414520Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6414769Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6415025Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6415275Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6415530Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6415805Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6416050Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6416302Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6416559Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6416809Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6417067Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6417311Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6417556Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6417808Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6418065Z STAGE:2022-11-23 02:16:42 1444:1444 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6418310Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6418567Z STAGE:2022-11-23 02:16:42 1443:1443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6418631Z ok (4.040s) 2022-11-23T02:32:19.6418636Z 2022-11-23T02:32:19.6418838Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6418914Z Ran 1 test in 4.040s 2022-11-23T02:32:19.6418918Z 2022-11-23T02:32:19.6418966Z OK 2022-11-23T02:32:19.6418970Z 2022-11-23T02:32:19.6419053Z Generating XML reports... 2022-11-23T02:32:19.6419400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021640.xml 2022-11-23T02:32:19.6419641Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6419928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6420051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6420340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6420475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6420481Z 2022-11-23T02:32:19.6420589Z Running tests... 2022-11-23T02:32:19.6420796Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6421498Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.519s) 2022-11-23T02:32:19.6421504Z 2022-11-23T02:32:19.6421697Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6421769Z Ran 1 test in 0.519s 2022-11-23T02:32:19.6421774Z 2022-11-23T02:32:19.6421839Z OK (skipped=1) 2022-11-23T02:32:19.6421844Z 2022-11-23T02:32:19.6421923Z Generating XML reports... 2022-11-23T02:32:19.6422267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021647.xml 2022-11-23T02:32:19.6422504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6422785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6422904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6423226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6423356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6423361Z 2022-11-23T02:32:19.6423427Z Running tests... 2022-11-23T02:32:19.6423625Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6423859Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1690 2022-11-23T02:32:19.6424011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1691 2022-11-23T02:32:19.6424200Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6424479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6424599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6424885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6425018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6425193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6425470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6425592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6425880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6426010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6426187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6426488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6426790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6426948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6427107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6427277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6427478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6427779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6428078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6428331Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6428581Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6428835Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6429085Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6429346Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6429603Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6429847Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6430131Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6430382Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6430638Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6430888Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6431143Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6431388Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6431631Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6431880Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6432136Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6432384Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6432643Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6432883Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6433123Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6433374Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6433627Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6433874Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6434130Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6434369Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6434612Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6434859Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6435114Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6435386Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6435640Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6435880Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6436175Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6436421Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6436677Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6436925Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6437178Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6437420Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6437661Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6437958Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6438215Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6438458Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6438711Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6438949Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6439191Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6439438Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6439694Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6439942Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6440198Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6440439Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6440682Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6440932Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6441181Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6441431Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6441685Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6441927Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6442168Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6442418Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6442675Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6442922Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6443178Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6443458Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6443697Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6443942Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6444201Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6444445Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6444700Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6444939Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6445178Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6445426Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6445682Z STAGE:2022-11-23 02:16:53 1691:1691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6445966Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6446225Z STAGE:2022-11-23 02:16:53 1690:1690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6446287Z ok (3.945s) 2022-11-23T02:32:19.6446292Z 2022-11-23T02:32:19.6446493Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6446556Z Ran 1 test in 3.946s 2022-11-23T02:32:19.6446568Z 2022-11-23T02:32:19.6446617Z OK 2022-11-23T02:32:19.6446621Z 2022-11-23T02:32:19.6446700Z Generating XML reports... 2022-11-23T02:32:19.6447048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021650.xml 2022-11-23T02:32:19.6447282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6447564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6447688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6447975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6448107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6448112Z 2022-11-23T02:32:19.6448180Z Running tests... 2022-11-23T02:32:19.6448378Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6448606Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1885 2022-11-23T02:32:19.6448758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1886 2022-11-23T02:32:19.6448953Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6449232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6449351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6449638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6449766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6449940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6450218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6450371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6450660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6450782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6450957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6451264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6451566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6451727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6451890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6451999Z skip: Skipped due to small world size. (3.739s) 2022-11-23T02:32:19.6452004Z 2022-11-23T02:32:19.6452205Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6452274Z Ran 1 test in 3.739s 2022-11-23T02:32:19.6452279Z 2022-11-23T02:32:19.6452344Z OK (skipped=1) 2022-11-23T02:32:19.6452383Z 2022-11-23T02:32:19.6452463Z Generating XML reports... 2022-11-23T02:32:19.6452807Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021657.xml 2022-11-23T02:32:19.6453042Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6453318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6453437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6453723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6453855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6453861Z 2022-11-23T02:32:19.6453926Z Running tests... 2022-11-23T02:32:19.6454121Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6454353Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2068 2022-11-23T02:32:19.6454504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2069 2022-11-23T02:32:19.6454696Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6454970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6455083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6455369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6455503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6455677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6455960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6456080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6456368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6456496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6456669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6456974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6457313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6457477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6457635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6458239Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:32:19.6458311Z warnings.warn( 2022-11-23T02:32:19.6458896Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:32:19.6458969Z warnings.warn( 2022-11-23T02:32:19.6459029Z ok (4.051s) 2022-11-23T02:32:19.6459034Z 2022-11-23T02:32:19.6459265Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6459337Z Ran 1 test in 4.052s 2022-11-23T02:32:19.6459342Z 2022-11-23T02:32:19.6459396Z OK 2022-11-23T02:32:19.6459401Z 2022-11-23T02:32:19.6459477Z Generating XML reports... 2022-11-23T02:32:19.6459823Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021704.xml 2022-11-23T02:32:19.6460058Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6460335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6460450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6460738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6460869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6460874Z 2022-11-23T02:32:19.6460942Z Running tests... 2022-11-23T02:32:19.6461139Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6461836Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.508s) 2022-11-23T02:32:19.6461842Z 2022-11-23T02:32:19.6462036Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6462109Z Ran 1 test in 0.509s 2022-11-23T02:32:19.6462113Z 2022-11-23T02:32:19.6462181Z OK (skipped=1) 2022-11-23T02:32:19.6462185Z 2022-11-23T02:32:19.6462264Z Generating XML reports... 2022-11-23T02:32:19.6462606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021711.xml 2022-11-23T02:32:19.6462844Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6463125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6463244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6463529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6463660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6463699Z 2022-11-23T02:32:19.6463767Z Running tests... 2022-11-23T02:32:19.6463966Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6464356Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:32:19.6464364Z 2022-11-23T02:32:19.6464561Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6464629Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6464634Z 2022-11-23T02:32:19.6464698Z OK (skipped=1) 2022-11-23T02:32:19.6464703Z 2022-11-23T02:32:19.6464782Z Generating XML reports... 2022-11-23T02:32:19.6465128Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021715.xml 2022-11-23T02:32:19.6465363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6465639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6465754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6466042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6466206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6466211Z 2022-11-23T02:32:19.6466281Z Running tests... 2022-11-23T02:32:19.6466481Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6466874Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:32:19.6466880Z 2022-11-23T02:32:19.6467077Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6467147Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6467154Z 2022-11-23T02:32:19.6467219Z OK (skipped=1) 2022-11-23T02:32:19.6467223Z 2022-11-23T02:32:19.6467300Z Generating XML reports... 2022-11-23T02:32:19.6467647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021718.xml 2022-11-23T02:32:19.6467884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6468163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6468285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6468574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6468706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6468710Z 2022-11-23T02:32:19.6468778Z Running tests... 2022-11-23T02:32:19.6468976Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6469210Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2425 2022-11-23T02:32:19.6469362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2426 2022-11-23T02:32:19.6469556Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6469833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6469952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6470229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6470362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6470539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6470852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6470972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6471257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6471389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6471562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6471869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6472171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6472334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6472495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6472676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1hpwxn9o 2022-11-23T02:32:19.6472904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1hpwxn9o/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6473083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyg_0p8eq 2022-11-23T02:32:19.6473273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyg_0p8eq/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6473436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6473600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6473662Z ok (6.449s) 2022-11-23T02:32:19.6473670Z 2022-11-23T02:32:19.6473873Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6473944Z Ran 1 test in 6.449s 2022-11-23T02:32:19.6473949Z 2022-11-23T02:32:19.6474003Z OK 2022-11-23T02:32:19.6474007Z 2022-11-23T02:32:19.6474078Z Generating XML reports... 2022-11-23T02:32:19.6474424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021721.xml 2022-11-23T02:32:19.6474658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6474939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6475058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6475346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6475479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6475486Z 2022-11-23T02:32:19.6475553Z Running tests... 2022-11-23T02:32:19.6475752Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6475997Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2618 2022-11-23T02:32:19.6476195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2619 2022-11-23T02:32:19.6476387Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6476664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6476785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6477068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6477243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6477415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6477695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6477819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6478110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6478242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6478418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6478720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6479016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6479180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6479383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6479570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyu634988 2022-11-23T02:32:19.6479758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyu634988/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6479936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvn6dt_cq 2022-11-23T02:32:19.6480123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvn6dt_cq/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6480287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6480453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6480615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6480776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6480841Z ok (6.465s) 2022-11-23T02:32:19.6480847Z 2022-11-23T02:32:19.6481050Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6481120Z Ran 1 test in 6.466s 2022-11-23T02:32:19.6481125Z 2022-11-23T02:32:19.6481180Z OK 2022-11-23T02:32:19.6481184Z 2022-11-23T02:32:19.6481262Z Generating XML reports... 2022-11-23T02:32:19.6481609Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021730.xml 2022-11-23T02:32:19.6481844Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6482124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6482246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6482536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6482662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6482675Z 2022-11-23T02:32:19.6482735Z Running tests... 2022-11-23T02:32:19.6482934Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6483635Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:32:19.6483686Z 2022-11-23T02:32:19.6483875Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6483945Z Ran 1 test in 0.509s 2022-11-23T02:32:19.6483950Z 2022-11-23T02:32:19.6484017Z OK (skipped=1) 2022-11-23T02:32:19.6484021Z 2022-11-23T02:32:19.6484099Z Generating XML reports... 2022-11-23T02:32:19.6484446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021740.xml 2022-11-23T02:32:19.6484684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6484961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6485082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6485369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6485500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6485505Z 2022-11-23T02:32:19.6485571Z Running tests... 2022-11-23T02:32:19.6485766Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6486522Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.526s) 2022-11-23T02:32:19.6486529Z 2022-11-23T02:32:19.6486726Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6486797Z Ran 1 test in 0.526s 2022-11-23T02:32:19.6486802Z 2022-11-23T02:32:19.6486866Z OK (skipped=1) 2022-11-23T02:32:19.6486871Z 2022-11-23T02:32:19.6486949Z Generating XML reports... 2022-11-23T02:32:19.6487294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021743.xml 2022-11-23T02:32:19.6487527Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6487809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6487930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6488218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6488350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6488355Z 2022-11-23T02:32:19.6488421Z Running tests... 2022-11-23T02:32:19.6488615Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6488963Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s) 2022-11-23T02:32:19.6488981Z 2022-11-23T02:32:19.6489170Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6489240Z Ran 1 test in 0.003s 2022-11-23T02:32:19.6489245Z 2022-11-23T02:32:19.6489313Z OK (skipped=1) 2022-11-23T02:32:19.6489319Z 2022-11-23T02:32:19.6489400Z Generating XML reports... 2022-11-23T02:32:19.6489742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021747.xml 2022-11-23T02:32:19.6489978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6490255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6490376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6490663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6490850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6490855Z 2022-11-23T02:32:19.6490919Z Running tests... 2022-11-23T02:32:19.6491113Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6491377Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2985 2022-11-23T02:32:19.6491528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2986 2022-11-23T02:32:19.6491721Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6491996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6492112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6492397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6492527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6492756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6493068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6493350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6493471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6493750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6493880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6494057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6494360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6494523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6494684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6494870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphl81cgh_ 2022-11-23T02:32:19.6495060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphl81cgh_/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6495238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mtmicy0 2022-11-23T02:32:19.6495427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mtmicy0/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6495490Z ok (4.243s) 2022-11-23T02:32:19.6495496Z 2022-11-23T02:32:19.6495696Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6495765Z Ran 1 test in 4.244s 2022-11-23T02:32:19.6495770Z 2022-11-23T02:32:19.6495824Z OK 2022-11-23T02:32:19.6495829Z 2022-11-23T02:32:19.6495909Z Generating XML reports... 2022-11-23T02:32:19.6496254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021750.xml 2022-11-23T02:32:19.6496488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6496768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6496888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6497175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6497335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6497340Z 2022-11-23T02:32:19.6497406Z Running tests... 2022-11-23T02:32:19.6497598Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6497827Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3168 2022-11-23T02:32:19.6497978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3169 2022-11-23T02:32:19.6498171Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6498451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6498573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6498859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6498990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6499161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6499469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6499590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6499873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6500001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6500171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6500475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6500779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6500939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6501098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6501281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6lt9pf5 2022-11-23T02:32:19.6501472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6lt9pf5/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6501652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cnavxay 2022-11-23T02:32:19.6501840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cnavxay/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6502006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6502165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6502227Z ok (6.351s) 2022-11-23T02:32:19.6502232Z 2022-11-23T02:32:19.6502433Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6502504Z Ran 1 test in 6.351s 2022-11-23T02:32:19.6502509Z 2022-11-23T02:32:19.6502564Z OK 2022-11-23T02:32:19.6502568Z 2022-11-23T02:32:19.6502649Z Generating XML reports... 2022-11-23T02:32:19.6502997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021757.xml 2022-11-23T02:32:19.6503230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6503510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6503664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6503953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6504083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6504088Z 2022-11-23T02:32:19.6504157Z Running tests... 2022-11-23T02:32:19.6504358Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6504720Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.004s) 2022-11-23T02:32:19.6504725Z 2022-11-23T02:32:19.6504919Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6504986Z Ran 1 test in 0.004s 2022-11-23T02:32:19.6504991Z 2022-11-23T02:32:19.6505054Z OK (skipped=1) 2022-11-23T02:32:19.6505058Z 2022-11-23T02:32:19.6505136Z Generating XML reports... 2022-11-23T02:32:19.6505477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021807.xml 2022-11-23T02:32:19.6505709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6506017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6506131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6506418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6506551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6506555Z 2022-11-23T02:32:19.6506622Z Running tests... 2022-11-23T02:32:19.6506822Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6507171Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s) 2022-11-23T02:32:19.6507178Z 2022-11-23T02:32:19.6507371Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6507444Z Ran 1 test in 0.003s 2022-11-23T02:32:19.6507449Z 2022-11-23T02:32:19.6507513Z OK (skipped=1) 2022-11-23T02:32:19.6507520Z 2022-11-23T02:32:19.6507599Z Generating XML reports... 2022-11-23T02:32:19.6507941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021810.xml 2022-11-23T02:32:19.6508175Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6508451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6508570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6508853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6508985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6508989Z 2022-11-23T02:32:19.6509055Z Running tests... 2022-11-23T02:32:19.6509246Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6509477Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3477 2022-11-23T02:32:19.6509631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3478 2022-11-23T02:32:19.6509820Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6510095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6510212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6510523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6510652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6510825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6511128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6511407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6511528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6511817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6511949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6512126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6512430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6512627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6512813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpz16qc1 2022-11-23T02:32:19.6513004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpz16qc1/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6513165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6513346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6byk7jj 2022-11-23T02:32:19.6513533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6byk7jj/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6514237Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6514928Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6515847Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1122.) 2022-11-23T02:32:19.6516016Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T02:32:19.6516233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6517129Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1122.) 2022-11-23T02:32:19.6517333Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T02:32:19.6517496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6518187Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6518868Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6519620Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6520299Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6520976Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6521651Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6522318Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6522991Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6523835Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6524541Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:32:19.6524645Z ok (4.040s) 2022-11-23T02:32:19.6524651Z 2022-11-23T02:32:19.6524866Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6524943Z Ran 1 test in 4.041s 2022-11-23T02:32:19.6524948Z 2022-11-23T02:32:19.6525007Z OK 2022-11-23T02:32:19.6525012Z 2022-11-23T02:32:19.6525094Z Generating XML reports... 2022-11-23T02:32:19.6525462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021812.xml 2022-11-23T02:32:19.6525711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6526006Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6526135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6526440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6526578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6526583Z 2022-11-23T02:32:19.6526678Z Running tests... 2022-11-23T02:32:19.6526890Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6527226Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.004s) 2022-11-23T02:32:19.6527231Z 2022-11-23T02:32:19.6527457Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6527546Z Ran 1 test in 0.004s 2022-11-23T02:32:19.6527552Z 2022-11-23T02:32:19.6527644Z OK (skipped=1) 2022-11-23T02:32:19.6527648Z 2022-11-23T02:32:19.6527733Z Generating XML reports... 2022-11-23T02:32:19.6528103Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021819.xml 2022-11-23T02:32:19.6528352Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6528652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6528782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6529091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6529227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6529232Z 2022-11-23T02:32:19.6529301Z Running tests... 2022-11-23T02:32:19.6529507Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6530036Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:32:19.6530044Z 2022-11-23T02:32:19.6530258Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6530336Z Ran 1 test in 0.003s 2022-11-23T02:32:19.6530341Z 2022-11-23T02:32:19.6530412Z OK (skipped=1) 2022-11-23T02:32:19.6530417Z 2022-11-23T02:32:19.6530502Z Generating XML reports... 2022-11-23T02:32:19.6530875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021822.xml 2022-11-23T02:32:19.6531130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6531430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6531560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6531872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6532052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6532057Z 2022-11-23T02:32:19.6532130Z Running tests... 2022-11-23T02:32:19.6532336Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6533106Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.511s) 2022-11-23T02:32:19.6533122Z 2022-11-23T02:32:19.6533339Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6533408Z Ran 1 test in 0.511s 2022-11-23T02:32:19.6533423Z 2022-11-23T02:32:19.6533486Z OK (skipped=1) 2022-11-23T02:32:19.6533491Z 2022-11-23T02:32:19.6533582Z Generating XML reports... 2022-11-23T02:32:19.6533955Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021825.xml 2022-11-23T02:32:19.6534213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6534550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6534682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6534996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6535138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6535143Z 2022-11-23T02:32:19.6535216Z Running tests... 2022-11-23T02:32:19.6535429Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6536294Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:32:19.6536304Z 2022-11-23T02:32:19.6536511Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6536587Z Ran 1 test in 0.510s 2022-11-23T02:32:19.6536593Z 2022-11-23T02:32:19.6536662Z OK (skipped=1) 2022-11-23T02:32:19.6536666Z 2022-11-23T02:32:19.6536751Z Generating XML reports... 2022-11-23T02:32:19.6537110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021829.xml 2022-11-23T02:32:19.6537357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6537655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6537783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6538089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6538232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6538237Z 2022-11-23T02:32:19.6538310Z Running tests... 2022-11-23T02:32:19.6538518Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6538786Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3896 2022-11-23T02:32:19.6538948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3897 2022-11-23T02:32:19.6539142Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6539471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6539596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6539898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6540033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6540215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6540603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6540722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6541007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6541139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6541313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6541663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6541969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6542132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6542292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6542461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6542630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6542932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6543228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6543415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikhd52ca 2022-11-23T02:32:19.6543607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikhd52ca/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6543786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8ibrmi9 2022-11-23T02:32:19.6543973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8ibrmi9/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6544132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6544298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6544463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6544626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6544687Z ok (6.850s) 2022-11-23T02:32:19.6544693Z 2022-11-23T02:32:19.6544894Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6544966Z Ran 1 test in 6.851s 2022-11-23T02:32:19.6544971Z 2022-11-23T02:32:19.6545025Z OK 2022-11-23T02:32:19.6545030Z 2022-11-23T02:32:19.6545107Z Generating XML reports... 2022-11-23T02:32:19.6545448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021832.xml 2022-11-23T02:32:19.6545683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6545963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6546119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6546410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6546547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6546552Z 2022-11-23T02:32:19.6546617Z Running tests... 2022-11-23T02:32:19.6546816Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6547056Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4095 2022-11-23T02:32:19.6547206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4096 2022-11-23T02:32:19.6547400Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6547684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6547801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6548112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6548243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6548417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6548696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6548816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6549103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6549233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6549404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6549706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6550011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6550172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6550368Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:32:19.6550524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6550716Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:32:19.6550901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps69th9nt 2022-11-23T02:32:19.6551094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps69th9nt/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6551279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrz4p4h2 2022-11-23T02:32:19.6551471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrz4p4h2/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6551639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6551805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6551971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6552136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6552375Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:32:19.6552568Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:32:19.6552766Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:32:19.6552957Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:32:19.6553119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6553279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6553439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6553599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6553799Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:32:19.6553992Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:32:19.6554223Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:32:19.6554413Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:32:19.6554575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6554736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6554897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6555059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6555122Z ok (7.864s) 2022-11-23T02:32:19.6555127Z 2022-11-23T02:32:19.6555332Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6555404Z Ran 1 test in 7.864s 2022-11-23T02:32:19.6555409Z 2022-11-23T02:32:19.6555465Z OK 2022-11-23T02:32:19.6555469Z 2022-11-23T02:32:19.6555551Z Generating XML reports... 2022-11-23T02:32:19.6555899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021842.xml 2022-11-23T02:32:19.6556229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6556528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6556649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6556935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6557071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6557076Z 2022-11-23T02:32:19.6557144Z Running tests... 2022-11-23T02:32:19.6557346Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6558054Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.512s) 2022-11-23T02:32:19.6558060Z 2022-11-23T02:32:19.6558255Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6558323Z Ran 1 test in 0.512s 2022-11-23T02:32:19.6558328Z 2022-11-23T02:32:19.6558393Z OK (skipped=1) 2022-11-23T02:32:19.6558398Z 2022-11-23T02:32:19.6558475Z Generating XML reports... 2022-11-23T02:32:19.6558867Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021853.xml 2022-11-23T02:32:19.6559101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6559383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6559502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6559789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6559919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6559924Z 2022-11-23T02:32:19.6559992Z Running tests... 2022-11-23T02:32:19.6560191Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6560429Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4346 2022-11-23T02:32:19.6560583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4347 2022-11-23T02:32:19.6560776Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6561090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6561211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6561500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6561629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6561796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6562076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6562196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6562480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6562616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6562789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6563093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6563393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6563551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6563961Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:32:19.6564122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6564528Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:32:19.6564712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga8kcp5m 2022-11-23T02:32:19.6564904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga8kcp5m/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6565120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwq94e2tn 2022-11-23T02:32:19.6565307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwq94e2tn/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6565477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6565643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6565844Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:32:19.6566038Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:32:19.6566252Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:32:19.6566465Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:32:19.6566703Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:32:19.6566970Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:32:19.6567206Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:32:19.6567429Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:32:19.6567594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6567753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6572170Z ok (6.448s) 2022-11-23T02:32:19.6572180Z 2022-11-23T02:32:19.6572430Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6572508Z Ran 1 test in 6.449s 2022-11-23T02:32:19.6572514Z 2022-11-23T02:32:19.6572569Z OK 2022-11-23T02:32:19.6572573Z 2022-11-23T02:32:19.6572660Z Generating XML reports... 2022-11-23T02:32:19.6573017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021857.xml 2022-11-23T02:32:19.6573254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6573543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6573663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6573955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6574088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6574094Z 2022-11-23T02:32:19.6574160Z Running tests... 2022-11-23T02:32:19.6574361Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6574631Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4539 2022-11-23T02:32:19.6574788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4540 2022-11-23T02:32:19.6574981Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6575266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6575380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6575756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6575888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6576066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6576349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6576471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6576760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6576891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6577064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6577371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6577675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6577874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6578037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6578228Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6578409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp225v2nf1 2022-11-23T02:32:19.6578599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp225v2nf1/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6578784Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6578964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimisms5x 2022-11-23T02:32:19.6579153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimisms5x/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6579321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6579487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6579650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6579814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6579870Z ok (6.743s) 2022-11-23T02:32:19.6579875Z 2022-11-23T02:32:19.6580080Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6580153Z Ran 1 test in 6.744s 2022-11-23T02:32:19.6580160Z 2022-11-23T02:32:19.6580217Z OK 2022-11-23T02:32:19.6580221Z 2022-11-23T02:32:19.6580300Z Generating XML reports... 2022-11-23T02:32:19.6580646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021906.xml 2022-11-23T02:32:19.6580883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6581162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6581281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6581566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6581694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6581699Z 2022-11-23T02:32:19.6581765Z Running tests... 2022-11-23T02:32:19.6581963Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6582272Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4786 2022-11-23T02:32:19.6582427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4787 2022-11-23T02:32:19.6582615Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6582891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6583008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6583291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6583421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6583595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6583874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6583992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6584311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6584443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6584610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6584914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6585215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6585380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6585540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6585733Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6585912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ip8h_e3 2022-11-23T02:32:19.6586102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ip8h_e3/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6586287Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6586464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2wta272 2022-11-23T02:32:19.6586651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2wta272/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6586816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6586979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6587141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6587302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6587363Z ok (6.848s) 2022-11-23T02:32:19.6587368Z 2022-11-23T02:32:19.6587570Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6587638Z Ran 1 test in 6.849s 2022-11-23T02:32:19.6587643Z 2022-11-23T02:32:19.6587696Z OK 2022-11-23T02:32:19.6587701Z 2022-11-23T02:32:19.6587777Z Generating XML reports... 2022-11-23T02:32:19.6588121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021916.xml 2022-11-23T02:32:19.6588393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6588671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6588784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6589071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6589200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6589205Z 2022-11-23T02:32:19.6589270Z Running tests... 2022-11-23T02:32:19.6589471Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6589784Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5033 2022-11-23T02:32:19.6589939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5034 2022-11-23T02:32:19.6590127Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6590439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6590559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6590845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6590973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6591145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6591448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6591727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6591844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6592134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6592264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6592434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6592736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6592895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6593055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6593245Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6593427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph43tbxs9 2022-11-23T02:32:19.6593612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph43tbxs9/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6593796Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6593971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmj8bsqqe 2022-11-23T02:32:19.6594156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmj8bsqqe/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6594322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6594485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6594702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6594865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6594927Z ok (6.760s) 2022-11-23T02:32:19.6594932Z 2022-11-23T02:32:19.6595136Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6595205Z Ran 1 test in 6.761s 2022-11-23T02:32:19.6595211Z 2022-11-23T02:32:19.6595265Z OK 2022-11-23T02:32:19.6595270Z 2022-11-23T02:32:19.6595345Z Generating XML reports... 2022-11-23T02:32:19.6595691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021926.xml 2022-11-23T02:32:19.6595928Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6596283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6596404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6596691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6596861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6596866Z 2022-11-23T02:32:19.6596932Z Running tests... 2022-11-23T02:32:19.6597132Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6597445Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5280 2022-11-23T02:32:19.6597596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5281 2022-11-23T02:32:19.6597782Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6598060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6598179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6598468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6598597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6598770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6599048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6599170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6599453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6599585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6599758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6600065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6600365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6600526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6600685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6600878Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6601068Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6601289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5nu_yiz 2022-11-23T02:32:19.6601483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5nu_yiz/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6601667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxf2botr 2022-11-23T02:32:19.6601861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxf2botr/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6602026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6602188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6602346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6602506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6602570Z ok (6.757s) 2022-11-23T02:32:19.6602575Z 2022-11-23T02:32:19.6602780Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6602850Z Ran 1 test in 6.758s 2022-11-23T02:32:19.6602855Z 2022-11-23T02:32:19.6602911Z OK 2022-11-23T02:32:19.6602916Z 2022-11-23T02:32:19.6603029Z Generating XML reports... 2022-11-23T02:32:19.6603381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021936.xml 2022-11-23T02:32:19.6603616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6603892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6604010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6604298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6604431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6604436Z 2022-11-23T02:32:19.6604503Z Running tests... 2022-11-23T02:32:19.6604700Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6605013Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5527 2022-11-23T02:32:19.6605164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5528 2022-11-23T02:32:19.6605355Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6605632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6605751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6606039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6606167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6606337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6606613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6606731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6607017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6607146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6607319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6607656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6607955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6608118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6608276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6608464Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6608646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt72fbmgy 2022-11-23T02:32:19.6608835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt72fbmgy/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6609019Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6609195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpto167kgv 2022-11-23T02:32:19.6609380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpto167kgv/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6609578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6609743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6609904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6610065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6610126Z ok (6.849s) 2022-11-23T02:32:19.6610131Z 2022-11-23T02:32:19.6610330Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6610403Z Ran 1 test in 6.849s 2022-11-23T02:32:19.6610409Z 2022-11-23T02:32:19.6610457Z OK 2022-11-23T02:32:19.6610466Z 2022-11-23T02:32:19.6610538Z Generating XML reports... 2022-11-23T02:32:19.6610882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021946.xml 2022-11-23T02:32:19.6611119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6611395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6611514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6611799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6611927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6611932Z 2022-11-23T02:32:19.6611998Z Running tests... 2022-11-23T02:32:19.6612198Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6612509Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5774 2022-11-23T02:32:19.6612661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5775 2022-11-23T02:32:19.6612850Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6613127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6613246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6613531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6613695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6613869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6614144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6614265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6614548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6614675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6614848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6615146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6615447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6615610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6615770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6615992Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6616175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfuynfud 2022-11-23T02:32:19.6616369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfuynfud/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6616554Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6616730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7f7rchr 2022-11-23T02:32:19.6616921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7f7rchr/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6617086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6617250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6617413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6617574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6617636Z ok (6.747s) 2022-11-23T02:32:19.6617642Z 2022-11-23T02:32:19.6617843Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6617911Z Ran 1 test in 6.747s 2022-11-23T02:32:19.6617916Z 2022-11-23T02:32:19.6617969Z OK 2022-11-23T02:32:19.6617974Z 2022-11-23T02:32:19.6618052Z Generating XML reports... 2022-11-23T02:32:19.6618394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021955.xml 2022-11-23T02:32:19.6618631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6618908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6619024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6619311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6619440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6619445Z 2022-11-23T02:32:19.6619510Z Running tests... 2022-11-23T02:32:19.6619708Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6620020Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6021 2022-11-23T02:32:19.6620206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6022 2022-11-23T02:32:19.6620396Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6620676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6620794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6621078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6621207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6621379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6621652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6621772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6622055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6622215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6622388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6622693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6622992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6623151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6623312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6623502Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6623682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp798zy9ve 2022-11-23T02:32:19.6623872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp798zy9ve/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6624053Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6624226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1c_1ir2e 2022-11-23T02:32:19.6624411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1c_1ir2e/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6624573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6624737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6624898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6625057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6625120Z ok (6.949s) 2022-11-23T02:32:19.6625125Z 2022-11-23T02:32:19.6625325Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6625394Z Ran 1 test in 6.950s 2022-11-23T02:32:19.6625399Z 2022-11-23T02:32:19.6625454Z OK 2022-11-23T02:32:19.6625459Z 2022-11-23T02:32:19.6625537Z Generating XML reports... 2022-11-23T02:32:19.6625880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022005.xml 2022-11-23T02:32:19.6626115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6626451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6626571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6626857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6626989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6626994Z 2022-11-23T02:32:19.6627058Z Running tests... 2022-11-23T02:32:19.6627256Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6627565Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6268 2022-11-23T02:32:19.6627716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6269 2022-11-23T02:32:19.6627904Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6628185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6628307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6628627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6628759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6628930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6629235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6629511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6629633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6629917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6630047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6630219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6630522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6630688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6630848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6631040Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6631226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wh3tnxs 2022-11-23T02:32:19.6631418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wh3tnxs/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6631606Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6631783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeb285njj 2022-11-23T02:32:19.6631970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeb285njj/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6632137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6632300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6632458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6632654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6632717Z ok (6.941s) 2022-11-23T02:32:19.6632722Z 2022-11-23T02:32:19.6632924Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6632996Z Ran 1 test in 6.942s 2022-11-23T02:32:19.6633001Z 2022-11-23T02:32:19.6633060Z OK 2022-11-23T02:32:19.6633064Z 2022-11-23T02:32:19.6633144Z Generating XML reports... 2022-11-23T02:32:19.6633492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022015.xml 2022-11-23T02:32:19.6633726Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6634005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6634125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6634415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6634548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6634553Z 2022-11-23T02:32:19.6634620Z Running tests... 2022-11-23T02:32:19.6634852Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6635162Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6515 2022-11-23T02:32:19.6635315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6516 2022-11-23T02:32:19.6635505Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6635782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6635901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6636235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6636366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6636540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6636839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6637122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6637242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6637531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6637667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6637841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6638144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6638307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6638470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6638661Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6638844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcg2c8r0o 2022-11-23T02:32:19.6639037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcg2c8r0o/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6639263Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6639440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jej0fry 2022-11-23T02:32:19.6639789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jej0fry/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6639964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6640139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6640310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6640481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6640546Z ok (7.049s) 2022-11-23T02:32:19.6640552Z 2022-11-23T02:32:19.6640763Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6640839Z Ran 1 test in 7.049s 2022-11-23T02:32:19.6640844Z 2022-11-23T02:32:19.6640895Z OK 2022-11-23T02:32:19.6640910Z 2022-11-23T02:32:19.6640985Z Generating XML reports... 2022-11-23T02:32:19.6641405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022025.xml 2022-11-23T02:32:19.6641658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6641953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6642080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6642383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6642522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6642530Z 2022-11-23T02:32:19.6642600Z Running tests... 2022-11-23T02:32:19.6642811Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6643136Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6762 2022-11-23T02:32:19.6643295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6763 2022-11-23T02:32:19.6643498Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6643794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6643920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6644222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6644364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6644546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6644873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6645168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6645296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6645599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6645736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6645917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6646271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6646444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6646617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6646822Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6647013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigt5k185 2022-11-23T02:32:19.6647215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigt5k185/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6647414Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6647604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewafssd3 2022-11-23T02:32:19.6647808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewafssd3/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6647983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6648192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6648366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6648541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6648611Z ok (6.848s) 2022-11-23T02:32:19.6648616Z 2022-11-23T02:32:19.6648826Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6648902Z Ran 1 test in 6.848s 2022-11-23T02:32:19.6648907Z 2022-11-23T02:32:19.6648967Z OK 2022-11-23T02:32:19.6648971Z 2022-11-23T02:32:19.6649053Z Generating XML reports... 2022-11-23T02:32:19.6649425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022035.xml 2022-11-23T02:32:19.6649671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6649973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6650100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6650396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6650536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6650540Z 2022-11-23T02:32:19.6650717Z Running tests... 2022-11-23T02:32:19.6650920Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6651192Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7009 2022-11-23T02:32:19.6651350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7010 2022-11-23T02:32:19.6651543Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6651821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6651939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6652227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6652356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6652530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6652844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6652963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6653247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6653378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6653550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6653859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6654163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6654326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6654488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6654682Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6654914Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6655101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbej67cu 2022-11-23T02:32:19.6655288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbej67cu/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6655473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6m4emio 2022-11-23T02:32:19.6655667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6m4emio/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6655830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6655997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6656160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6656323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6656392Z ok (6.698s) 2022-11-23T02:32:19.6656397Z 2022-11-23T02:32:19.6656603Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6656675Z Ran 1 test in 6.698s 2022-11-23T02:32:19.6656680Z 2022-11-23T02:32:19.6656733Z OK 2022-11-23T02:32:19.6656738Z 2022-11-23T02:32:19.6656819Z Generating XML reports... 2022-11-23T02:32:19.6657165Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022045.xml 2022-11-23T02:32:19.6657404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6657684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6657802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6658092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6658222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6658227Z 2022-11-23T02:32:19.6658292Z Running tests... 2022-11-23T02:32:19.6658489Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6658753Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7256 2022-11-23T02:32:19.6658905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7257 2022-11-23T02:32:19.6659089Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6659399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6659517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6659809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6659940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6660111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6660416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6660691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6660809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6661094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6661223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6661429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6661737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6661899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6662059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6662251Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6662433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n0xhzs8 2022-11-23T02:32:19.6662626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n0xhzs8/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6662812Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6662986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilfntl9u 2022-11-23T02:32:19.6663176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilfntl9u/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6663342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6663506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6663663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6663830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6663891Z ok (6.847s) 2022-11-23T02:32:19.6663896Z 2022-11-23T02:32:19.6664130Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6664199Z Ran 1 test in 6.847s 2022-11-23T02:32:19.6664203Z 2022-11-23T02:32:19.6664260Z OK 2022-11-23T02:32:19.6664267Z 2022-11-23T02:32:19.6664347Z Generating XML reports... 2022-11-23T02:32:19.6664690Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022055.xml 2022-11-23T02:32:19.6664922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6665202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6665323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6665609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6665775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6665779Z 2022-11-23T02:32:19.6665848Z Running tests... 2022-11-23T02:32:19.6666046Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6666379Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:32:19.6666384Z 2022-11-23T02:32:19.6666579Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6666648Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6666653Z 2022-11-23T02:32:19.6666720Z OK (skipped=1) 2022-11-23T02:32:19.6666724Z 2022-11-23T02:32:19.6666802Z Generating XML reports... 2022-11-23T02:32:19.6667143Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022105.xml 2022-11-23T02:32:19.6667383Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6667655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6667774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6668093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6668229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6668233Z 2022-11-23T02:32:19.6668299Z Running tests... 2022-11-23T02:32:19.6668497Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6668723Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7561 2022-11-23T02:32:19.6668876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7562 2022-11-23T02:32:19.6669071Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6669346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6669466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6669751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6669882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6670057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6670356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6670632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6670751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6671037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6671166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6671337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6671638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6671798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6671955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6672132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzoji68z 2022-11-23T02:32:19.6672360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzoji68z/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6672538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_350p4cx 2022-11-23T02:32:19.6672727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_350p4cx/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6672788Z ok (6.440s) 2022-11-23T02:32:19.6672793Z 2022-11-23T02:32:19.6672992Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6673060Z Ran 1 test in 6.440s 2022-11-23T02:32:19.6673065Z 2022-11-23T02:32:19.6673120Z OK 2022-11-23T02:32:19.6673124Z 2022-11-23T02:32:19.6673201Z Generating XML reports... 2022-11-23T02:32:19.6673545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022108.xml 2022-11-23T02:32:19.6673777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6674058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6674177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6674495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6674625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6674630Z 2022-11-23T02:32:19.6674695Z Running tests... 2022-11-23T02:32:19.6674895Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6675136Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7747 2022-11-23T02:32:19.6675286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7748 2022-11-23T02:32:19.6675481Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6675756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6675878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6676209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6676342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6676519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6676797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6676917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6677207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6677337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6677510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6677819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6678119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6678283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6678443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6678624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhocfm4h 2022-11-23T02:32:19.6678855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhocfm4h/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6679034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrpgyd59 2022-11-23T02:32:19.6679224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrpgyd59/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6679390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6679553Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6679861Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:32:19.6679974Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:32:19.6680270Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:32:19.6680376Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:32:19.6680434Z ok (6.482s) 2022-11-23T02:32:19.6680439Z 2022-11-23T02:32:19.6680638Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6680707Z Ran 1 test in 6.482s 2022-11-23T02:32:19.6680712Z 2022-11-23T02:32:19.6680766Z OK 2022-11-23T02:32:19.6680771Z 2022-11-23T02:32:19.6680886Z Generating XML reports... 2022-11-23T02:32:19.6681232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022118.xml 2022-11-23T02:32:19.6681464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6681741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6681860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6682145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6682279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6682284Z 2022-11-23T02:32:19.6682351Z Running tests... 2022-11-23T02:32:19.6682550Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6682789Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7940 2022-11-23T02:32:19.6682941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7941 2022-11-23T02:32:19.6683132Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6683408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6683527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6683816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6683946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6684119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6684399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6684511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6684797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6684929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6685104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6685409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6685742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6685900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6686090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppr36edn2 2022-11-23T02:32:19.6686282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppr36edn2/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6686442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6686621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7umh7qii 2022-11-23T02:32:19.6686810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7umh7qii/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6686978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6687145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6687205Z ok (4.061s) 2022-11-23T02:32:19.6687210Z 2022-11-23T02:32:19.6687407Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6687506Z Ran 1 test in 4.061s 2022-11-23T02:32:19.6687512Z 2022-11-23T02:32:19.6687567Z OK 2022-11-23T02:32:19.6687571Z 2022-11-23T02:32:19.6687648Z Generating XML reports... 2022-11-23T02:32:19.6687993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022127.xml 2022-11-23T02:32:19.6688224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6688502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6688618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6688904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6689035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6689040Z 2022-11-23T02:32:19.6689109Z Running tests... 2022-11-23T02:32:19.6689306Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6689536Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8181 2022-11-23T02:32:19.6689687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8182 2022-11-23T02:32:19.6689881Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6690155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6690276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6690559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6690693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6690866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6691170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6691445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6691565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6691850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6692026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6692195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6692501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6692660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6692817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6692994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc345of9 2022-11-23T02:32:19.6693182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc345of9/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6693363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5q69lrs 2022-11-23T02:32:19.6693550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5q69lrs/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6693713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6693876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6693969Z ok (6.450s) 2022-11-23T02:32:19.6693975Z 2022-11-23T02:32:19.6694179Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6694250Z Ran 1 test in 6.451s 2022-11-23T02:32:19.6694255Z 2022-11-23T02:32:19.6694308Z OK 2022-11-23T02:32:19.6694312Z 2022-11-23T02:32:19.6694391Z Generating XML reports... 2022-11-23T02:32:19.6694736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022134.xml 2022-11-23T02:32:19.6694971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6695254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6695374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6695660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6695791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6695797Z 2022-11-23T02:32:19.6695863Z Running tests... 2022-11-23T02:32:19.6696057Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6696413Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:32:19.6696418Z 2022-11-23T02:32:19.6696614Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6696681Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6696688Z 2022-11-23T02:32:19.6696752Z OK (skipped=1) 2022-11-23T02:32:19.6696757Z 2022-11-23T02:32:19.6696830Z Generating XML reports... 2022-11-23T02:32:19.6697169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022144.xml 2022-11-23T02:32:19.6697404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6697682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6697799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6698084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6698212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6698217Z 2022-11-23T02:32:19.6698283Z Running tests... 2022-11-23T02:32:19.6698512Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6698856Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:32:19.6698861Z 2022-11-23T02:32:19.6699055Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6699122Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6699128Z 2022-11-23T02:32:19.6699191Z OK (skipped=1) 2022-11-23T02:32:19.6699196Z 2022-11-23T02:32:19.6699272Z Generating XML reports... 2022-11-23T02:32:19.6699610Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022147.xml 2022-11-23T02:32:19.6699843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6700119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6700239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6700522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6700649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6700653Z 2022-11-23T02:32:19.6700751Z Running tests... 2022-11-23T02:32:19.6700945Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6701320Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:32:19.6701325Z 2022-11-23T02:32:19.6701516Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6701580Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6701585Z 2022-11-23T02:32:19.6701649Z OK (skipped=1) 2022-11-23T02:32:19.6701653Z 2022-11-23T02:32:19.6701734Z Generating XML reports... 2022-11-23T02:32:19.6702073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022150.xml 2022-11-23T02:32:19.6702305Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6702579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6702696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6702977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6703103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6703108Z 2022-11-23T02:32:19.6703176Z Running tests... 2022-11-23T02:32:19.6703370Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6703730Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:32:19.6703737Z 2022-11-23T02:32:19.6703933Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6704000Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6704005Z 2022-11-23T02:32:19.6704069Z OK (skipped=1) 2022-11-23T02:32:19.6704074Z 2022-11-23T02:32:19.6704150Z Generating XML reports... 2022-11-23T02:32:19.6704491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022153.xml 2022-11-23T02:32:19.6704725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6704999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6705118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6705439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6705567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6705572Z 2022-11-23T02:32:19.6705639Z Running tests... 2022-11-23T02:32:19.6705832Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6706149Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:32:19.6706154Z 2022-11-23T02:32:19.6706346Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6706414Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6706419Z 2022-11-23T02:32:19.6706482Z OK (skipped=1) 2022-11-23T02:32:19.6706486Z 2022-11-23T02:32:19.6706563Z Generating XML reports... 2022-11-23T02:32:19.6706903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022155.xml 2022-11-23T02:32:19.6707137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6707414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6707561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6707848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6707975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6707980Z 2022-11-23T02:32:19.6708047Z Running tests... 2022-11-23T02:32:19.6708242Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6708471Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8664 2022-11-23T02:32:19.6708623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8665 2022-11-23T02:32:19.6708814Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6709094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6709212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6709494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6709625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6709798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6710073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6710187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6710470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6710598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6710772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6711073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6711377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6711536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6711693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6711875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwo8boxgq 2022-11-23T02:32:19.6712103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwo8boxgq/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6712279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2k25oh78 2022-11-23T02:32:19.6712469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2k25oh78/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6713056Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:32:19.6713666Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:32:19.6713833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6713997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6714057Z ok (6.459s) 2022-11-23T02:32:19.6714062Z 2022-11-23T02:32:19.6714262Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6714332Z Ran 1 test in 6.459s 2022-11-23T02:32:19.6714339Z 2022-11-23T02:32:19.6714394Z OK 2022-11-23T02:32:19.6714399Z 2022-11-23T02:32:19.6714475Z Generating XML reports... 2022-11-23T02:32:19.6714817Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022158.xml 2022-11-23T02:32:19.6715050Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6715328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6715444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6715730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6715859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6715864Z 2022-11-23T02:32:19.6715931Z Running tests... 2022-11-23T02:32:19.6716183Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6716913Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:32:19.6716919Z 2022-11-23T02:32:19.6717116Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6717186Z Ran 1 test in 0.510s 2022-11-23T02:32:19.6717191Z 2022-11-23T02:32:19.6717256Z OK (skipped=1) 2022-11-23T02:32:19.6717260Z 2022-11-23T02:32:19.6717339Z Generating XML reports... 2022-11-23T02:32:19.6717676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022208.xml 2022-11-23T02:32:19.6717910Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6718230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6718351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6718637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6718767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6718772Z 2022-11-23T02:32:19.6718836Z Running tests... 2022-11-23T02:32:19.6719033Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6719380Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:32:19.6719386Z 2022-11-23T02:32:19.6719577Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6719647Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6719652Z 2022-11-23T02:32:19.6719717Z OK (skipped=1) 2022-11-23T02:32:19.6719722Z 2022-11-23T02:32:19.6719799Z Generating XML reports... 2022-11-23T02:32:19.6720174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022211.xml 2022-11-23T02:32:19.6720408Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6720688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6720804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6721091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6721220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6721227Z 2022-11-23T02:32:19.6721292Z Running tests... 2022-11-23T02:32:19.6721487Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6721826Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:32:19.6721831Z 2022-11-23T02:32:19.6722027Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6722090Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6722101Z 2022-11-23T02:32:19.6722159Z OK (skipped=1) 2022-11-23T02:32:19.6722171Z 2022-11-23T02:32:19.6722242Z Generating XML reports... 2022-11-23T02:32:19.6722582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022214.xml 2022-11-23T02:32:19.6722818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6723096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6723217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6723501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6723632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6723637Z 2022-11-23T02:32:19.6723702Z Running tests... 2022-11-23T02:32:19.6723898Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6724131Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9031 2022-11-23T02:32:19.6724282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9032 2022-11-23T02:32:19.6724471Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6724782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6724901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6725184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6725312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6725484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6725760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6725876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6726160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6726291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6726459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6726767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6727100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6727261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6727419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6727603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigznul19 2022-11-23T02:32:19.6727795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigznul19/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6727976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc77b9web 2022-11-23T02:32:19.6728162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc77b9web/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6728225Z ok (4.252s) 2022-11-23T02:32:19.6728230Z 2022-11-23T02:32:19.6728431Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6728501Z Ran 1 test in 4.252s 2022-11-23T02:32:19.6728505Z 2022-11-23T02:32:19.6728559Z OK 2022-11-23T02:32:19.6728563Z 2022-11-23T02:32:19.6728640Z Generating XML reports... 2022-11-23T02:32:19.6728987Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022217.xml 2022-11-23T02:32:19.6729219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6729497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6729617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6729903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6730031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6730036Z 2022-11-23T02:32:19.6730103Z Running tests... 2022-11-23T02:32:19.6730300Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6731008Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.524s) 2022-11-23T02:32:19.6731013Z 2022-11-23T02:32:19.6731210Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6731372Z Ran 1 test in 0.524s 2022-11-23T02:32:19.6731377Z 2022-11-23T02:32:19.6731436Z OK (skipped=1) 2022-11-23T02:32:19.6731448Z 2022-11-23T02:32:19.6731519Z Generating XML reports... 2022-11-23T02:32:19.6731865Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022225.xml 2022-11-23T02:32:19.6732101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6732380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6732498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6732784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6732912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6732919Z 2022-11-23T02:32:19.6732985Z Running tests... 2022-11-23T02:32:19.6733182Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6733531Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:32:19.6733568Z 2022-11-23T02:32:19.6733761Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6733829Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6733834Z 2022-11-23T02:32:19.6733899Z OK (skipped=1) 2022-11-23T02:32:19.6733903Z 2022-11-23T02:32:19.6733982Z Generating XML reports... 2022-11-23T02:32:19.6734319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022228.xml 2022-11-23T02:32:19.6734552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6734827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6734948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6735232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6735362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6735367Z 2022-11-23T02:32:19.6735433Z Running tests... 2022-11-23T02:32:19.6735628Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6736337Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.524s) 2022-11-23T02:32:19.6736345Z 2022-11-23T02:32:19.6736537Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6736605Z Ran 1 test in 0.524s 2022-11-23T02:32:19.6736610Z 2022-11-23T02:32:19.6736668Z OK (skipped=1) 2022-11-23T02:32:19.6736678Z 2022-11-23T02:32:19.6736750Z Generating XML reports... 2022-11-23T02:32:19.6737087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022231.xml 2022-11-23T02:32:19.6737318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6737594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6737712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6737993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6738155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6738160Z 2022-11-23T02:32:19.6738224Z Running tests... 2022-11-23T02:32:19.6738421Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6738660Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9388 2022-11-23T02:32:19.6738813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9389 2022-11-23T02:32:19.6739003Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6739280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6739399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6739684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6739815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6739991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6740319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6740441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6740725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6740855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6741026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6741324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6741629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6741790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6741950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6742134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy3capl_a 2022-11-23T02:32:19.6742327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy3capl_a/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6742503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7_dnx7g 2022-11-23T02:32:19.6742691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7_dnx7g/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6742944Z STAGE:2022-11-23 02:22:37 9389:9389 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6743195Z STAGE:2022-11-23 02:22:37 9388:9388 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6743360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6743525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:32:19.6743781Z STAGE:2022-11-23 02:22:37 9389:9389 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6744028Z STAGE:2022-11-23 02:22:37 9388:9388 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6744290Z STAGE:2022-11-23 02:22:37 9389:9389 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6744551Z STAGE:2022-11-23 02:22:37 9388:9388 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6744718Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:32:19.6744916Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:32:19.6745162Z STAGE:2022-11-23 02:22:37 9388:9388 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6745414Z STAGE:2022-11-23 02:22:37 9388:9388 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6745676Z STAGE:2022-11-23 02:22:37 9388:9388 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6745737Z ok (4.445s) 2022-11-23T02:32:19.6745742Z 2022-11-23T02:32:19.6745943Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6746007Z Ran 1 test in 4.445s 2022-11-23T02:32:19.6746012Z 2022-11-23T02:32:19.6746067Z OK 2022-11-23T02:32:19.6746071Z 2022-11-23T02:32:19.6746149Z Generating XML reports... 2022-11-23T02:32:19.6746491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022235.xml 2022-11-23T02:32:19.6746726Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6747009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6747160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6747450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6747580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6747585Z 2022-11-23T02:32:19.6747650Z Running tests... 2022-11-23T02:32:19.6747846Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6748079Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9585 2022-11-23T02:32:19.6748233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9586 2022-11-23T02:32:19.6748422Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6748703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6748822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6749104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6749233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6749404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6749680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6749800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6750083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6750207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6750381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6750686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6750985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6751145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6751304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6751488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppeqnodll 2022-11-23T02:32:19.6751712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppeqnodll/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6751891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpy16v_k 2022-11-23T02:32:19.6752081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpy16v_k/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6752142Z ok (4.350s) 2022-11-23T02:32:19.6752147Z 2022-11-23T02:32:19.6752348Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6752418Z Ran 1 test in 4.350s 2022-11-23T02:32:19.6752423Z 2022-11-23T02:32:19.6752475Z OK 2022-11-23T02:32:19.6752479Z 2022-11-23T02:32:19.6752555Z Generating XML reports... 2022-11-23T02:32:19.6752899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022242.xml 2022-11-23T02:32:19.6753133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6753412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6753532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6753865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6753996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6754001Z 2022-11-23T02:32:19.6754067Z Running tests... 2022-11-23T02:32:19.6754260Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6754499Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9772 2022-11-23T02:32:19.6754650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9773 2022-11-23T02:32:19.6754843Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6755120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6755239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6755527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6755653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6755823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6756147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6756264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6756553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6756681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6756853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6757155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6757455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6757615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6757773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6757951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy3oajsa7 2022-11-23T02:32:19.6758184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy3oajsa7/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6758359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8536yn0 2022-11-23T02:32:19.6758546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8536yn0/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6758602Z ok (4.242s) 2022-11-23T02:32:19.6758614Z 2022-11-23T02:32:19.6758810Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6758880Z Ran 1 test in 4.242s 2022-11-23T02:32:19.6758885Z 2022-11-23T02:32:19.6758940Z OK 2022-11-23T02:32:19.6758944Z 2022-11-23T02:32:19.6759024Z Generating XML reports... 2022-11-23T02:32:19.6759366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022250.xml 2022-11-23T02:32:19.6759599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6759879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6759999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6760320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6760449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6760454Z 2022-11-23T02:32:19.6760519Z Running tests... 2022-11-23T02:32:19.6760716Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6761426Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:32:19.6761434Z 2022-11-23T02:32:19.6761626Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6761696Z Ran 1 test in 0.510s 2022-11-23T02:32:19.6761701Z 2022-11-23T02:32:19.6761766Z OK (skipped=1) 2022-11-23T02:32:19.6761770Z 2022-11-23T02:32:19.6761850Z Generating XML reports... 2022-11-23T02:32:19.6762189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022257.xml 2022-11-23T02:32:19.6762421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6762695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6762814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6763099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6763231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6763236Z 2022-11-23T02:32:19.6763302Z Running tests... 2022-11-23T02:32:19.6763491Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6764189Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.528s) 2022-11-23T02:32:19.6764195Z 2022-11-23T02:32:19.6764389Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6764458Z Ran 1 test in 0.528s 2022-11-23T02:32:19.6764462Z 2022-11-23T02:32:19.6764522Z OK (skipped=1) 2022-11-23T02:32:19.6764530Z 2022-11-23T02:32:19.6764639Z Generating XML reports... 2022-11-23T02:32:19.6764980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022300.xml 2022-11-23T02:32:19.6765215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6765496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6765616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6765906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6766035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6766039Z 2022-11-23T02:32:19.6766104Z Running tests... 2022-11-23T02:32:19.6766297Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6767052Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.513s) 2022-11-23T02:32:19.6767060Z 2022-11-23T02:32:19.6767256Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6767325Z Ran 1 test in 0.513s 2022-11-23T02:32:19.6767330Z 2022-11-23T02:32:19.6767394Z OK (skipped=1) 2022-11-23T02:32:19.6767398Z 2022-11-23T02:32:19.6767475Z Generating XML reports... 2022-11-23T02:32:19.6767813Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022304.xml 2022-11-23T02:32:19.6768047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6768328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6768446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6768731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6768862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6768867Z 2022-11-23T02:32:19.6768932Z Running tests... 2022-11-23T02:32:19.6769129Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6769493Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:32:19.6769498Z 2022-11-23T02:32:19.6769693Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6769763Z Ran 1 test in 0.003s 2022-11-23T02:32:19.6769770Z 2022-11-23T02:32:19.6769829Z OK (skipped=1) 2022-11-23T02:32:19.6769839Z 2022-11-23T02:32:19.6769910Z Generating XML reports... 2022-11-23T02:32:19.6770248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022307.xml 2022-11-23T02:32:19.6770487Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6770762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6770880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6771165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6771294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6771299Z 2022-11-23T02:32:19.6771364Z Running tests... 2022-11-23T02:32:19.6771559Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6771826Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10187 2022-11-23T02:32:19.6771978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10188 2022-11-23T02:32:19.6772168Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6772444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6772561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6772844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6772977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6773151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6773429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6773547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6773866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6773996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6774163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6774467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6774764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6774925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6775083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6775369Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:32:19.6775560Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:32:19.6775742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbu0crigk 2022-11-23T02:32:19.6775932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbu0crigk/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6776213Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:32:19.6776400Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:32:19.6776578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmj9rufn 2022-11-23T02:32:19.6776769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmj9rufn/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6776829Z ok (4.239s) 2022-11-23T02:32:19.6776834Z 2022-11-23T02:32:19.6777032Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6777105Z Ran 1 test in 4.239s 2022-11-23T02:32:19.6777111Z 2022-11-23T02:32:19.6777165Z OK 2022-11-23T02:32:19.6777169Z 2022-11-23T02:32:19.6777245Z Generating XML reports... 2022-11-23T02:32:19.6777586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022310.xml 2022-11-23T02:32:19.6777819Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6778094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6778247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6778528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6778658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6778668Z 2022-11-23T02:32:19.6778730Z Running tests... 2022-11-23T02:32:19.6778928Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6779156Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10370 2022-11-23T02:32:19.6779307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10371 2022-11-23T02:32:19.6779498Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6779772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6779894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6780179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6780340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6780514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6780819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6781095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6781211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6781493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6781624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6781794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6782101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6782263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6782421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6782591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6782760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6783060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6783355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6783417Z ok (3.942s) 2022-11-23T02:32:19.6783421Z 2022-11-23T02:32:19.6783620Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6783691Z Ran 1 test in 3.943s 2022-11-23T02:32:19.6783696Z 2022-11-23T02:32:19.6783751Z OK 2022-11-23T02:32:19.6783755Z 2022-11-23T02:32:19.6783832Z Generating XML reports... 2022-11-23T02:32:19.6784175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022317.xml 2022-11-23T02:32:19.6784410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6784685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6784805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6785128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6785258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6785263Z 2022-11-23T02:32:19.6785328Z Running tests... 2022-11-23T02:32:19.6785529Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6785752Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10559 2022-11-23T02:32:19.6785903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10560 2022-11-23T02:32:19.6786092Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6786367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6786485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6786768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6786896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6787100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6787375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6787493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6787777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6787905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6788077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6788384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6788683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6788842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6789001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6789170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6789338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6789638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6789933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6789994Z ok (3.840s) 2022-11-23T02:32:19.6789999Z 2022-11-23T02:32:19.6790196Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6790268Z Ran 1 test in 3.841s 2022-11-23T02:32:19.6790273Z 2022-11-23T02:32:19.6790329Z OK 2022-11-23T02:32:19.6790333Z 2022-11-23T02:32:19.6790411Z Generating XML reports... 2022-11-23T02:32:19.6790754Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022324.xml 2022-11-23T02:32:19.6790987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6791263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6791380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6791716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6791839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6791851Z 2022-11-23T02:32:19.6791911Z Running tests... 2022-11-23T02:32:19.6792111Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6792817Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:32:19.6792828Z 2022-11-23T02:32:19.6793017Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6793087Z Ran 1 test in 0.510s 2022-11-23T02:32:19.6793093Z 2022-11-23T02:32:19.6793157Z OK (skipped=1) 2022-11-23T02:32:19.6793161Z 2022-11-23T02:32:19.6793240Z Generating XML reports... 2022-11-23T02:32:19.6793577Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022331.xml 2022-11-23T02:32:19.6793848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6794128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6794247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6794531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6794659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6794664Z 2022-11-23T02:32:19.6794730Z Running tests... 2022-11-23T02:32:19.6794930Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6795268Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:32:19.6795273Z 2022-11-23T02:32:19.6795470Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6795540Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6795544Z 2022-11-23T02:32:19.6795607Z OK (skipped=1) 2022-11-23T02:32:19.6795611Z 2022-11-23T02:32:19.6795689Z Generating XML reports... 2022-11-23T02:32:19.6796023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022335.xml 2022-11-23T02:32:19.6796307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6796583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6796703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6796987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6797112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6797124Z 2022-11-23T02:32:19.6797184Z Running tests... 2022-11-23T02:32:19.6797381Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6797617Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10864 2022-11-23T02:32:19.6797769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10865 2022-11-23T02:32:19.6797961Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6798240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6798403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6798688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6798819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6798991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6799265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6799382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6799665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6799793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6799967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6800271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6800607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6800769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6800926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6800986Z ok (4.036s) 2022-11-23T02:32:19.6800991Z 2022-11-23T02:32:19.6801188Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6801251Z Ran 1 test in 4.037s 2022-11-23T02:32:19.6801262Z 2022-11-23T02:32:19.6801310Z OK 2022-11-23T02:32:19.6801316Z 2022-11-23T02:32:19.6801392Z Generating XML reports... 2022-11-23T02:32:19.6801733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022338.xml 2022-11-23T02:32:19.6801964Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6802243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6802361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6802646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6802774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6802779Z 2022-11-23T02:32:19.6802845Z Running tests... 2022-11-23T02:32:19.6803044Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6803266Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11047 2022-11-23T02:32:19.6803417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11048 2022-11-23T02:32:19.6803611Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6803889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6804006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6804292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6804422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6804595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6804934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6805210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6805330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6805620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6805743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6805910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6806212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6806371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6806629Z STAGE:2022-11-23 02:23:47 11048:11048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6806785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6807071Z STAGE:2022-11-23 02:23:47 11047:11047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6807327Z STAGE:2022-11-23 02:23:47 11048:11048 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6807592Z STAGE:2022-11-23 02:23:47 11048:11048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6807844Z STAGE:2022-11-23 02:23:47 11047:11047 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6808104Z STAGE:2022-11-23 02:23:47 11047:11047 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6808348Z STAGE:2022-11-23 02:23:47 11048:11048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6808590Z STAGE:2022-11-23 02:23:47 11047:11047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6808838Z STAGE:2022-11-23 02:23:47 11047:11047 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6809096Z STAGE:2022-11-23 02:23:47 11047:11047 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6809345Z STAGE:2022-11-23 02:23:47 11048:11048 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6809605Z STAGE:2022-11-23 02:23:47 11048:11048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6809664Z ok (3.949s) 2022-11-23T02:32:19.6809670Z 2022-11-23T02:32:19.6809867Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6809937Z Ran 1 test in 3.949s 2022-11-23T02:32:19.6809942Z 2022-11-23T02:32:19.6809995Z OK 2022-11-23T02:32:19.6809999Z 2022-11-23T02:32:19.6810080Z Generating XML reports... 2022-11-23T02:32:19.6810422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022345.xml 2022-11-23T02:32:19.6810653Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6810932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6811050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6811334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6811464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6811469Z 2022-11-23T02:32:19.6811533Z Running tests... 2022-11-23T02:32:19.6811734Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6811958Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11236 2022-11-23T02:32:19.6812146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11237 2022-11-23T02:32:19.6812337Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6812614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6812730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6813011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6813139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6813311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6813589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6813705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6813986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6814149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6814322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6814624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6814926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6815082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6815241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6815301Z ok (3.728s) 2022-11-23T02:32:19.6815305Z 2022-11-23T02:32:19.6815502Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6815570Z Ran 1 test in 3.729s 2022-11-23T02:32:19.6815575Z 2022-11-23T02:32:19.6815630Z OK 2022-11-23T02:32:19.6815634Z 2022-11-23T02:32:19.6815710Z Generating XML reports... 2022-11-23T02:32:19.6816052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022352.xml 2022-11-23T02:32:19.6816286Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6816563Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6816680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6816963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6817092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6817097Z 2022-11-23T02:32:19.6817162Z Running tests... 2022-11-23T02:32:19.6817359Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6817537Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s) 2022-11-23T02:32:19.6817543Z 2022-11-23T02:32:19.6817737Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6817806Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6817811Z 2022-11-23T02:32:19.6817873Z OK (skipped=1) 2022-11-23T02:32:19.6817877Z 2022-11-23T02:32:19.6817956Z Generating XML reports... 2022-11-23T02:32:19.6818297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022359.xml 2022-11-23T02:32:19.6818563Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6818831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6818949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6819236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6819363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6819368Z 2022-11-23T02:32:19.6819434Z Running tests... 2022-11-23T02:32:19.6819628Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6819853Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11477 2022-11-23T02:32:19.6820004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11478 2022-11-23T02:32:19.6820196Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6820469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6820622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6820910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6825679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6825891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6826208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6826328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6826627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6826757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6826933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6827233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6827531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6827688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6827843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6828008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6828175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6828471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6828764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6829020Z STAGE:2022-11-23 02:24:04 11477:11477 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6829265Z STAGE:2022-11-23 02:24:04 11478:11478 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6829520Z STAGE:2022-11-23 02:24:04 11478:11478 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6829767Z STAGE:2022-11-23 02:24:04 11477:11477 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6830122Z STAGE:2022-11-23 02:24:04 11477:11477 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6830385Z STAGE:2022-11-23 02:24:04 11478:11478 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6830631Z STAGE:2022-11-23 02:24:04 11478:11478 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6830871Z STAGE:2022-11-23 02:24:04 11477:11477 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6831117Z STAGE:2022-11-23 02:24:04 11477:11477 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6831376Z STAGE:2022-11-23 02:24:04 11477:11477 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6831628Z STAGE:2022-11-23 02:24:04 11478:11478 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6831888Z STAGE:2022-11-23 02:24:04 11478:11478 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6831951Z ok (4.341s) 2022-11-23T02:32:19.6831957Z 2022-11-23T02:32:19.6832156Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6832225Z Ran 1 test in 4.341s 2022-11-23T02:32:19.6832231Z 2022-11-23T02:32:19.6832285Z OK 2022-11-23T02:32:19.6832290Z 2022-11-23T02:32:19.6832402Z Generating XML reports... 2022-11-23T02:32:19.6832751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022401.xml 2022-11-23T02:32:19.6832988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6833268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6833388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6833676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6833808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6833813Z 2022-11-23T02:32:19.6833881Z Running tests... 2022-11-23T02:32:19.6834078Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6834306Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11672 2022-11-23T02:32:19.6834459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11673 2022-11-23T02:32:19.6834648Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6834927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6835045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6835328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6835458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6835631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6835933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6836283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6836400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6836688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6836818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6836988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6837332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6837493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6837652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6837757Z skip: Skipped due to small world size. (3.841s) 2022-11-23T02:32:19.6837763Z 2022-11-23T02:32:19.6837961Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6838030Z Ran 1 test in 3.841s 2022-11-23T02:32:19.6838035Z 2022-11-23T02:32:19.6838099Z OK (skipped=1) 2022-11-23T02:32:19.6838103Z 2022-11-23T02:32:19.6838180Z Generating XML reports... 2022-11-23T02:32:19.6838525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022409.xml 2022-11-23T02:32:19.6838764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6839047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6839163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6839482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6839614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6839619Z 2022-11-23T02:32:19.6839684Z Running tests... 2022-11-23T02:32:19.6839881Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6840107Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11855 2022-11-23T02:32:19.6840259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11856 2022-11-23T02:32:19.6840449Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6840725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6840845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6841129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6841256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6841427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6841703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6841819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6842107Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6842233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6842408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6842709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6843006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6843165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6843319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6843381Z ok (3.739s) 2022-11-23T02:32:19.6843423Z 2022-11-23T02:32:19.6843624Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6843692Z Ran 1 test in 3.740s 2022-11-23T02:32:19.6843697Z 2022-11-23T02:32:19.6843750Z OK 2022-11-23T02:32:19.6843754Z 2022-11-23T02:32:19.6843830Z Generating XML reports... 2022-11-23T02:32:19.6844177Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022416.xml 2022-11-23T02:32:19.6844411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6844688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6844803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6845086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6845214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6845221Z 2022-11-23T02:32:19.6845286Z Running tests... 2022-11-23T02:32:19.6845480Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6846230Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.528s) 2022-11-23T02:32:19.6846237Z 2022-11-23T02:32:19.6846433Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6846501Z Ran 1 test in 0.528s 2022-11-23T02:32:19.6846505Z 2022-11-23T02:32:19.6846569Z OK (skipped=1) 2022-11-23T02:32:19.6846573Z 2022-11-23T02:32:19.6846651Z Generating XML reports... 2022-11-23T02:32:19.6846993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022423.xml 2022-11-23T02:32:19.6847229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6847506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6847628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6847913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6848043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6848048Z 2022-11-23T02:32:19.6848109Z Running tests... 2022-11-23T02:32:19.6848307Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6848528Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12096 2022-11-23T02:32:19.6848683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12097 2022-11-23T02:32:19.6848873Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6849151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6849269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6849552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6849681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6849853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6850129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6850278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6850561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6850688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6850863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6851166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6851470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6851628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6851788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6851958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6852124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6852453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6852752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6852808Z ok (3.756s) 2022-11-23T02:32:19.6852817Z 2022-11-23T02:32:19.6853009Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6853077Z Ran 1 test in 3.757s 2022-11-23T02:32:19.6853081Z 2022-11-23T02:32:19.6853133Z OK 2022-11-23T02:32:19.6853138Z 2022-11-23T02:32:19.6853215Z Generating XML reports... 2022-11-23T02:32:19.6853557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022426.xml 2022-11-23T02:32:19.6853792Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6854067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6854184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6854469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6854596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6854601Z 2022-11-23T02:32:19.6854666Z Running tests... 2022-11-23T02:32:19.6854861Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6855080Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12285 2022-11-23T02:32:19.6855234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12286 2022-11-23T02:32:19.6855424Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6855703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6855819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6856104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6856232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6856401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6856703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6857013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6857125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6857409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6857538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6857708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6858009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6858165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6858323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6858387Z ok (3.840s) 2022-11-23T02:32:19.6858391Z 2022-11-23T02:32:19.6858589Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6858657Z Ran 1 test in 3.841s 2022-11-23T02:32:19.6858662Z 2022-11-23T02:32:19.6858715Z OK 2022-11-23T02:32:19.6858719Z 2022-11-23T02:32:19.6858797Z Generating XML reports... 2022-11-23T02:32:19.6859168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022433.xml 2022-11-23T02:32:19.6859402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6859682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6859799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6860081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6860210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6860215Z 2022-11-23T02:32:19.6860279Z Running tests... 2022-11-23T02:32:19.6860475Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6860693Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12468 2022-11-23T02:32:19.6860839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12469 2022-11-23T02:32:19.6861028Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6861305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6861422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6861702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6861831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6862002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6862278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6862395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6862680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6862807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6862976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6863277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6863614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6863774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6863934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6863997Z ok (3.947s) 2022-11-23T02:32:19.6864001Z 2022-11-23T02:32:19.6864197Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6864264Z Ran 1 test in 3.948s 2022-11-23T02:32:19.6864269Z 2022-11-23T02:32:19.6864321Z OK 2022-11-23T02:32:19.6864326Z 2022-11-23T02:32:19.6864402Z Generating XML reports... 2022-11-23T02:32:19.6864743Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022440.xml 2022-11-23T02:32:19.6864972Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6865248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6865365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6865684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6865813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6865818Z 2022-11-23T02:32:19.6865883Z Running tests... 2022-11-23T02:32:19.6866079Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6866312Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12651 2022-11-23T02:32:19.6866462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12652 2022-11-23T02:32:19.6866654Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6866929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6867046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6867331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6867461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6867630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6867906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6868021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6868303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6868433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6868604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6868907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6869206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6869365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6869518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6869686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6869891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6870188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6870484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6870545Z ok (3.948s) 2022-11-23T02:32:19.6870550Z 2022-11-23T02:32:19.6870747Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6870816Z Ran 1 test in 3.949s 2022-11-23T02:32:19.6870821Z 2022-11-23T02:32:19.6870873Z OK 2022-11-23T02:32:19.6870877Z 2022-11-23T02:32:19.6870953Z Generating XML reports... 2022-11-23T02:32:19.6871297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022447.xml 2022-11-23T02:32:19.6871531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6871812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6871929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6872247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6872377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6872383Z 2022-11-23T02:32:19.6872448Z Running tests... 2022-11-23T02:32:19.6872646Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6872872Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12840 2022-11-23T02:32:19.6873023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12841 2022-11-23T02:32:19.6873216Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6873490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6873607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6873887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6874018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6874190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6874468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6874585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6874871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6875001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6875170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6875474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6875775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6875935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6876141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6876311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6876519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6876820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6877118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6877178Z ok (3.839s) 2022-11-23T02:32:19.6877184Z 2022-11-23T02:32:19.6877381Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6877449Z Ran 1 test in 3.840s 2022-11-23T02:32:19.6877454Z 2022-11-23T02:32:19.6877506Z OK 2022-11-23T02:32:19.6877511Z 2022-11-23T02:32:19.6877586Z Generating XML reports... 2022-11-23T02:32:19.6877931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022454.xml 2022-11-23T02:32:19.6878167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6878441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6878559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6878882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6879012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6879016Z 2022-11-23T02:32:19.6879081Z Running tests... 2022-11-23T02:32:19.6879278Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6879607Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:32:19.6879613Z 2022-11-23T02:32:19.6879804Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6879875Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6879880Z 2022-11-23T02:32:19.6879943Z OK (skipped=1) 2022-11-23T02:32:19.6879947Z 2022-11-23T02:32:19.6880025Z Generating XML reports... 2022-11-23T02:32:19.6880364Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022501.xml 2022-11-23T02:32:19.6880599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6880872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6880988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6881273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6881401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6881406Z 2022-11-23T02:32:19.6881470Z Running tests... 2022-11-23T02:32:19.6881667Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6881884Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13087 2022-11-23T02:32:19.6882038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13088 2022-11-23T02:32:19.6882228Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6882503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6882616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6882898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6883024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6883229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6883505Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6883619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6883905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6884031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6884202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6884504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6884800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6884960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6885117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6885178Z ok (4.427s) 2022-11-23T02:32:19.6885183Z 2022-11-23T02:32:19.6885412Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6885481Z Ran 1 test in 4.428s 2022-11-23T02:32:19.6885486Z 2022-11-23T02:32:19.6885539Z OK 2022-11-23T02:32:19.6885544Z 2022-11-23T02:32:19.6885620Z Generating XML reports... 2022-11-23T02:32:19.6885962Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022504.xml 2022-11-23T02:32:19.6886196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6886469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6886588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6886866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6886995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6887002Z 2022-11-23T02:32:19.6887067Z Running tests... 2022-11-23T02:32:19.6887264Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6887478Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13270 2022-11-23T02:32:19.6887631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13271 2022-11-23T02:32:19.6887821Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6888097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6888217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6888500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6888633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6888805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6889107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6889383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6889498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6889782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6889964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6890134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6890442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6890601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6890758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6890819Z ok (3.841s) 2022-11-23T02:32:19.6890824Z 2022-11-23T02:32:19.6891023Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6891086Z Ran 1 test in 3.841s 2022-11-23T02:32:19.6891090Z 2022-11-23T02:32:19.6891144Z OK 2022-11-23T02:32:19.6891148Z 2022-11-23T02:32:19.6891229Z Generating XML reports... 2022-11-23T02:32:19.6891574Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022511.xml 2022-11-23T02:32:19.6891808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6892121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6892240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6892525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6892654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6892659Z 2022-11-23T02:32:19.6892725Z Running tests... 2022-11-23T02:32:19.6892923Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6893158Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13453 2022-11-23T02:32:19.6893310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13454 2022-11-23T02:32:19.6893502Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6893777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6893893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6894177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6894304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6894475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6894752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6894867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6895156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6895283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6895452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6895755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6896055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6896213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6896520Z STAGE:2022-11-23 02:25:20 13454:13454 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6896677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6896931Z STAGE:2022-11-23 02:25:20 13453:13453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6897183Z STAGE:2022-11-23 02:25:20 13454:13454 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6897445Z STAGE:2022-11-23 02:25:20 13454:13454 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6897699Z STAGE:2022-11-23 02:25:21 13453:13453 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6897961Z STAGE:2022-11-23 02:25:21 13453:13453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6898022Z ok (4.040s) 2022-11-23T02:32:19.6898027Z 2022-11-23T02:32:19.6898224Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6898295Z Ran 1 test in 4.040s 2022-11-23T02:32:19.6898300Z 2022-11-23T02:32:19.6898353Z OK 2022-11-23T02:32:19.6898358Z 2022-11-23T02:32:19.6898435Z Generating XML reports... 2022-11-23T02:32:19.6898810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022518.xml 2022-11-23T02:32:19.6899047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6899327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6899448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6899732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6899859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6899871Z 2022-11-23T02:32:19.6899932Z Running tests... 2022-11-23T02:32:19.6900128Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6900359Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13642 2022-11-23T02:32:19.6900514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13643 2022-11-23T02:32:19.6900706Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6900979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6901096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6901380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6901510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6901686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6901986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6902264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6902384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6902669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6902797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6902968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6903267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6903458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6903715Z STAGE:2022-11-23 02:25:27 13643:13643 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6903875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6904128Z STAGE:2022-11-23 02:25:27 13642:13642 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.6904384Z STAGE:2022-11-23 02:25:27 13643:13643 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6904641Z STAGE:2022-11-23 02:25:27 13643:13643 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6904894Z STAGE:2022-11-23 02:25:27 13642:13642 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.6905157Z STAGE:2022-11-23 02:25:27 13642:13642 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.6905217Z ok (3.739s) 2022-11-23T02:32:19.6905222Z 2022-11-23T02:32:19.6905419Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6905486Z Ran 1 test in 3.739s 2022-11-23T02:32:19.6905491Z 2022-11-23T02:32:19.6905576Z OK 2022-11-23T02:32:19.6905581Z 2022-11-23T02:32:19.6905659Z Generating XML reports... 2022-11-23T02:32:19.6906004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022525.xml 2022-11-23T02:32:19.6906237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6906515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6906633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6906919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6907045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6907050Z 2022-11-23T02:32:19.6907116Z Running tests... 2022-11-23T02:32:19.6907314Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6907664Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:32:19.6907669Z 2022-11-23T02:32:19.6907861Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6907928Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6907933Z 2022-11-23T02:32:19.6907996Z OK (skipped=1) 2022-11-23T02:32:19.6908001Z 2022-11-23T02:32:19.6908075Z Generating XML reports... 2022-11-23T02:32:19.6908413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022532.xml 2022-11-23T02:32:19.6908648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6908922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6909041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6909326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6909455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6909460Z 2022-11-23T02:32:19.6909525Z Running tests... 2022-11-23T02:32:19.6909718Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6910087Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:32:19.6910125Z 2022-11-23T02:32:19.6910320Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6910388Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6910392Z 2022-11-23T02:32:19.6910456Z OK (skipped=1) 2022-11-23T02:32:19.6910460Z 2022-11-23T02:32:19.6910537Z Generating XML reports... 2022-11-23T02:32:19.6910880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022535.xml 2022-11-23T02:32:19.6911113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6911392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6911508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6911791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6911922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6911926Z 2022-11-23T02:32:19.6911989Z Running tests... 2022-11-23T02:32:19.6912184Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6912458Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13947 2022-11-23T02:32:19.6912611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13948 2022-11-23T02:32:19.6912799Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6913078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6913195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6913474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6913604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6913776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6914054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6914171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6914456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6914587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6914760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6915059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6915363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6915521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6915680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6915783Z skip: Skipped due to small world size. (3.741s) 2022-11-23T02:32:19.6915788Z 2022-11-23T02:32:19.6915983Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6916110Z Ran 1 test in 3.742s 2022-11-23T02:32:19.6916115Z 2022-11-23T02:32:19.6916179Z OK (skipped=1) 2022-11-23T02:32:19.6916183Z 2022-11-23T02:32:19.6916259Z Generating XML reports... 2022-11-23T02:32:19.6916598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022538.xml 2022-11-23T02:32:19.6916871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6917148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6917265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6917549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6917672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6917681Z 2022-11-23T02:32:19.6917741Z Running tests... 2022-11-23T02:32:19.6917937Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6918170Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14130 2022-11-23T02:32:19.6918323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14131 2022-11-23T02:32:19.6918515Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6918790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6918943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6919228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6919355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6919526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6919803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6919922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6920207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6920334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6920507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6920810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6921104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6921264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6921421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6921581Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:32:19.6921647Z ok (5.843s) 2022-11-23T02:32:19.6921652Z 2022-11-23T02:32:19.6921849Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6921913Z Ran 1 test in 5.843s 2022-11-23T02:32:19.6921918Z 2022-11-23T02:32:19.6921972Z OK 2022-11-23T02:32:19.6921977Z 2022-11-23T02:32:19.6922054Z Generating XML reports... 2022-11-23T02:32:19.6922396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022544.xml 2022-11-23T02:32:19.6922627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6922905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6923022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6923307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6923473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6923478Z 2022-11-23T02:32:19.6923543Z Running tests... 2022-11-23T02:32:19.6923742Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6923991Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14313 2022-11-23T02:32:19.6924143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14314 2022-11-23T02:32:19.6924334Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6924609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6924727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6925013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6925140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6925309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6925616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6925733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6926017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6926141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6926312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6926611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6926913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6927074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6927234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6927403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6927571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6927867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6928159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6928310Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-11-23T02:32:19.6928429Z No ranks successfully processed in monitoredBarrier. 2022-11-23T02:32:19.6928585Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-11-23T02:32:19.6928647Z ok (3.841s) 2022-11-23T02:32:19.6928652Z 2022-11-23T02:32:19.6928848Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6928916Z Ran 1 test in 3.842s 2022-11-23T02:32:19.6928921Z 2022-11-23T02:32:19.6928975Z OK 2022-11-23T02:32:19.6928979Z 2022-11-23T02:32:19.6929055Z Generating XML reports... 2022-11-23T02:32:19.6929395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022553.xml 2022-11-23T02:32:19.6929628Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6929938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6930055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6930335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6930467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6930472Z 2022-11-23T02:32:19.6930538Z Running tests... 2022-11-23T02:32:19.6930734Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6930977Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14502 2022-11-23T02:32:19.6931126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14503 2022-11-23T02:32:19.6931315Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6931589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6931705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6932019Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6932149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6932318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6932596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6932713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6932995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6933125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6933296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6933601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6933897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6934058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6934215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6934384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.6934551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.6934844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6935139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.6935302Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-11-23T02:32:19.6935362Z ok (3.927s) 2022-11-23T02:32:19.6935367Z 2022-11-23T02:32:19.6935565Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6935634Z Ran 1 test in 3.927s 2022-11-23T02:32:19.6935639Z 2022-11-23T02:32:19.6935692Z OK 2022-11-23T02:32:19.6935696Z 2022-11-23T02:32:19.6935776Z Generating XML reports... 2022-11-23T02:32:19.6936121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022600.xml 2022-11-23T02:32:19.6936390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6936666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6936784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6937068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6937196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6937201Z 2022-11-23T02:32:19.6937265Z Running tests... 2022-11-23T02:32:19.6937462Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6937703Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14691 2022-11-23T02:32:19.6937854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14692 2022-11-23T02:32:19.6938045Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6938320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6938470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6938758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6938887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6939054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6939329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6939447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6939732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6939859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6940032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6940332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6940631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6940788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6940947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6941052Z skip: Skipped due to small world size. (3.742s) 2022-11-23T02:32:19.6941060Z 2022-11-23T02:32:19.6941257Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6941326Z Ran 1 test in 3.742s 2022-11-23T02:32:19.6941331Z 2022-11-23T02:32:19.6941395Z OK (skipped=1) 2022-11-23T02:32:19.6941399Z 2022-11-23T02:32:19.6941475Z Generating XML reports... 2022-11-23T02:32:19.6941816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022607.xml 2022-11-23T02:32:19.6942050Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6942324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6942442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6942723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6942886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6942891Z 2022-11-23T02:32:19.6942957Z Running tests... 2022-11-23T02:32:19.6943149Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6943459Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:32:19.6943469Z 2022-11-23T02:32:19.6943657Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6943725Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6943730Z 2022-11-23T02:32:19.6943796Z OK (skipped=1) 2022-11-23T02:32:19.6943801Z 2022-11-23T02:32:19.6943879Z Generating XML reports... 2022-11-23T02:32:19.6944220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022614.xml 2022-11-23T02:32:19.6944455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6944733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6944852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6945244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6945374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6945378Z 2022-11-23T02:32:19.6945443Z Running tests... 2022-11-23T02:32:19.6945638Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6945947Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:32:19.6945952Z 2022-11-23T02:32:19.6946144Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6946212Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6946219Z 2022-11-23T02:32:19.6946282Z OK (skipped=1) 2022-11-23T02:32:19.6946286Z 2022-11-23T02:32:19.6946363Z Generating XML reports... 2022-11-23T02:32:19.6946702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022617.xml 2022-11-23T02:32:19.6946937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6947212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6947330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6947614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6947738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6947749Z 2022-11-23T02:32:19.6947809Z Running tests... 2022-11-23T02:32:19.6948006Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6948316Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:32:19.6948321Z 2022-11-23T02:32:19.6948514Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6948583Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6948587Z 2022-11-23T02:32:19.6948652Z OK (skipped=1) 2022-11-23T02:32:19.6948657Z 2022-11-23T02:32:19.6948733Z Generating XML reports... 2022-11-23T02:32:19.6949076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022620.xml 2022-11-23T02:32:19.6949308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6949583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6949700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6950022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6950150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6950155Z 2022-11-23T02:32:19.6950219Z Running tests... 2022-11-23T02:32:19.6950413Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6950714Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:32:19.6950719Z 2022-11-23T02:32:19.6950912Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6950978Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6950983Z 2022-11-23T02:32:19.6951045Z OK (skipped=1) 2022-11-23T02:32:19.6951050Z 2022-11-23T02:32:19.6951126Z Generating XML reports... 2022-11-23T02:32:19.6951464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022623.xml 2022-11-23T02:32:19.6951698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6951970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6952122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6952409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6952537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6952541Z 2022-11-23T02:32:19.6952607Z Running tests... 2022-11-23T02:32:19.6952800Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6953015Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s) 2022-11-23T02:32:19.6953022Z 2022-11-23T02:32:19.6953213Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6953280Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6953285Z 2022-11-23T02:32:19.6953347Z OK (skipped=1) 2022-11-23T02:32:19.6953352Z 2022-11-23T02:32:19.6953426Z Generating XML reports... 2022-11-23T02:32:19.6953766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022626.xml 2022-11-23T02:32:19.6953996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6954269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6954386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6954671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6954799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6954804Z 2022-11-23T02:32:19.6954867Z Running tests... 2022-11-23T02:32:19.6955059Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6955237Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:32:19.6955243Z 2022-11-23T02:32:19.6955434Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6955500Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6955505Z 2022-11-23T02:32:19.6955563Z OK (skipped=1) 2022-11-23T02:32:19.6955571Z 2022-11-23T02:32:19.6955642Z Generating XML reports... 2022-11-23T02:32:19.6955975Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022629.xml 2022-11-23T02:32:19.6956258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6956576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6956692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6956976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6957103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6957108Z 2022-11-23T02:32:19.6957172Z Running tests... 2022-11-23T02:32:19.6957365Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6957555Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:32:19.6957560Z 2022-11-23T02:32:19.6957751Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6957818Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6957825Z 2022-11-23T02:32:19.6957887Z OK (skipped=1) 2022-11-23T02:32:19.6957892Z 2022-11-23T02:32:19.6957968Z Generating XML reports... 2022-11-23T02:32:19.6958308Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022631.xml 2022-11-23T02:32:19.6958577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6958855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6958973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6959256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6959382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6959387Z 2022-11-23T02:32:19.6959453Z Running tests... 2022-11-23T02:32:19.6959646Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6959865Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:32:19.6959874Z 2022-11-23T02:32:19.6960062Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6960132Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6960137Z 2022-11-23T02:32:19.6960199Z OK (skipped=1) 2022-11-23T02:32:19.6960203Z 2022-11-23T02:32:19.6960279Z Generating XML reports... 2022-11-23T02:32:19.6960615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022634.xml 2022-11-23T02:32:19.6960846Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6961121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6961240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6961521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6961647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6961652Z 2022-11-23T02:32:19.6961717Z Running tests... 2022-11-23T02:32:19.6961910Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6962171Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15338 2022-11-23T02:32:19.6962322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15339 2022-11-23T02:32:19.6962509Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6962783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6962933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6963217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6963347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6963518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6963793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6963908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6964187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6964316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6964491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6964796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6965131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6965292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6965451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6965510Z ok (3.639s) 2022-11-23T02:32:19.6965516Z 2022-11-23T02:32:19.6965714Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6965782Z Ran 1 test in 3.639s 2022-11-23T02:32:19.6965787Z 2022-11-23T02:32:19.6965840Z OK 2022-11-23T02:32:19.6965844Z 2022-11-23T02:32:19.6965924Z Generating XML reports... 2022-11-23T02:32:19.6966263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022637.xml 2022-11-23T02:32:19.6966495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6966774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6966891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6967173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6967301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6967306Z 2022-11-23T02:32:19.6967372Z Running tests... 2022-11-23T02:32:19.6967567Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6967821Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15521 2022-11-23T02:32:19.6967977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15522 2022-11-23T02:32:19.6968168Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6968438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6968559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6968842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6968969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6969140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6969451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6969567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6969855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6969982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6970152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6970453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6970750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6970910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6971069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6971130Z ok (3.941s) 2022-11-23T02:32:19.6971135Z 2022-11-23T02:32:19.6971332Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6971400Z Ran 1 test in 3.941s 2022-11-23T02:32:19.6971442Z 2022-11-23T02:32:19.6971496Z OK 2022-11-23T02:32:19.6971500Z 2022-11-23T02:32:19.6971576Z Generating XML reports... 2022-11-23T02:32:19.6971919Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022644.xml 2022-11-23T02:32:19.6972150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6972425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6972538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6972828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6972956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6972961Z 2022-11-23T02:32:19.6973025Z Running tests... 2022-11-23T02:32:19.6973223Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6973419Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:32:19.6973424Z 2022-11-23T02:32:19.6973616Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6973683Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6973688Z 2022-11-23T02:32:19.6973753Z OK (skipped=1) 2022-11-23T02:32:19.6973758Z 2022-11-23T02:32:19.6973834Z Generating XML reports... 2022-11-23T02:32:19.6974175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022651.xml 2022-11-23T02:32:19.6974410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6974684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6974803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6975087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6975217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6975221Z 2022-11-23T02:32:19.6975285Z Running tests... 2022-11-23T02:32:19.6975479Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6975691Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:32:19.6975729Z 2022-11-23T02:32:19.6975922Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6975989Z Ran 1 test in 0.002s 2022-11-23T02:32:19.6975994Z 2022-11-23T02:32:19.6976058Z OK (skipped=1) 2022-11-23T02:32:19.6976062Z 2022-11-23T02:32:19.6976139Z Generating XML reports... 2022-11-23T02:32:19.6976474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022654.xml 2022-11-23T02:32:19.6976705Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6976982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6977101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6977385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6977515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6977521Z 2022-11-23T02:32:19.6977586Z Running tests... 2022-11-23T02:32:19.6977777Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6978519Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.500s) 2022-11-23T02:32:19.6978526Z 2022-11-23T02:32:19.6978725Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6978795Z Ran 1 test in 0.500s 2022-11-23T02:32:19.6978800Z 2022-11-23T02:32:19.6978864Z OK (skipped=1) 2022-11-23T02:32:19.6978869Z 2022-11-23T02:32:19.6978945Z Generating XML reports... 2022-11-23T02:32:19.6979284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022657.xml 2022-11-23T02:32:19.6979517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6979793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6979910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6980194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6980323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6980327Z 2022-11-23T02:32:19.6980392Z Running tests... 2022-11-23T02:32:19.6980586Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6980828Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15878 2022-11-23T02:32:19.6980982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15879 2022-11-23T02:32:19.6981174Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6981452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6981570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6981849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6981978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6982875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6983150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6983307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6983594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6983721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6983892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6984195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6984496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6984654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6984808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6984994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpghgns2 2022-11-23T02:32:19.6985187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpghgns2/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6985394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fmfn4ri 2022-11-23T02:32:19.6985585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fmfn4ri/_remote_module_non_scriptable.py 2022-11-23T02:32:19.6985647Z ok (6.362s) 2022-11-23T02:32:19.6985652Z 2022-11-23T02:32:19.6985853Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6985920Z Ran 1 test in 6.363s 2022-11-23T02:32:19.6985925Z 2022-11-23T02:32:19.6985977Z OK 2022-11-23T02:32:19.6985981Z 2022-11-23T02:32:19.6986057Z Generating XML reports... 2022-11-23T02:32:19.6986399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022700.xml 2022-11-23T02:32:19.6986632Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6986907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6987028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6987313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6987442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6987446Z 2022-11-23T02:32:19.6987510Z Running tests... 2022-11-23T02:32:19.6987704Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6987941Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16071 2022-11-23T02:32:19.6988096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16072 2022-11-23T02:32:19.6988287Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6988561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6988678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6988961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6989089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6989261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6989537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6989690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6989975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6990103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6990276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6990582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6990885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6991044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6991199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6991257Z ok (4.441s) 2022-11-23T02:32:19.6991266Z 2022-11-23T02:32:19.6991458Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6991526Z Ran 1 test in 4.442s 2022-11-23T02:32:19.6991531Z 2022-11-23T02:32:19.6991584Z OK 2022-11-23T02:32:19.6991588Z 2022-11-23T02:32:19.6991664Z Generating XML reports... 2022-11-23T02:32:19.6992037Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022710.xml 2022-11-23T02:32:19.6992273Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6992549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6992667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6992952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6993085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6993090Z 2022-11-23T02:32:19.6993156Z Running tests... 2022-11-23T02:32:19.6993352Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6993601Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16258 2022-11-23T02:32:19.6993753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16259 2022-11-23T02:32:19.6993943Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.6994218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6994336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6994621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6994751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6994922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.6995197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6995309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6995596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6995722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6995893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.6996234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6996589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.6996747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.6996909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.6996970Z ok (4.342s) 2022-11-23T02:32:19.6996975Z 2022-11-23T02:32:19.6997170Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6997238Z Ran 1 test in 4.343s 2022-11-23T02:32:19.6997243Z 2022-11-23T02:32:19.6997295Z OK 2022-11-23T02:32:19.6997299Z 2022-11-23T02:32:19.6997376Z Generating XML reports... 2022-11-23T02:32:19.6997716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022717.xml 2022-11-23T02:32:19.6997948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.6998225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.6998341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.6998661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.6998792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.6998797Z 2022-11-23T02:32:19.6998862Z Running tests... 2022-11-23T02:32:19.6999060Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.6999767Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.523s) 2022-11-23T02:32:19.6999775Z 2022-11-23T02:32:19.6999966Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7000033Z Ran 1 test in 0.524s 2022-11-23T02:32:19.7000038Z 2022-11-23T02:32:19.7000104Z OK (skipped=1) 2022-11-23T02:32:19.7000109Z 2022-11-23T02:32:19.7000186Z Generating XML reports... 2022-11-23T02:32:19.7000518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022725.xml 2022-11-23T02:32:19.7000748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7001021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7001138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7001421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7001550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7001555Z 2022-11-23T02:32:19.7001620Z Running tests... 2022-11-23T02:32:19.7001814Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7002539Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:32:19.7002544Z 2022-11-23T02:32:19.7002737Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7002805Z Ran 1 test in 0.510s 2022-11-23T02:32:19.7002842Z 2022-11-23T02:32:19.7002908Z OK (skipped=1) 2022-11-23T02:32:19.7002913Z 2022-11-23T02:32:19.7002990Z Generating XML reports... 2022-11-23T02:32:19.7003328Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022728.xml 2022-11-23T02:32:19.7003562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7003835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7003951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7004235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7004363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7004368Z 2022-11-23T02:32:19.7004433Z Running tests... 2022-11-23T02:32:19.7004625Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7005409Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75052 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.511s) 2022-11-23T02:32:19.7005416Z 2022-11-23T02:32:19.7005610Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7005676Z Ran 1 test in 0.511s 2022-11-23T02:32:19.7005681Z 2022-11-23T02:32:19.7005744Z OK (skipped=1) 2022-11-23T02:32:19.7005748Z 2022-11-23T02:32:19.7005824Z Generating XML reports... 2022-11-23T02:32:19.7006162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022732.xml 2022-11-23T02:32:19.7006394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7006673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7006785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7007070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7007200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7007204Z 2022-11-23T02:32:19.7007270Z Running tests... 2022-11-23T02:32:19.7007465Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7008222Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75139 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.523s) 2022-11-23T02:32:19.7008230Z 2022-11-23T02:32:19.7008424Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7008493Z Ran 1 test in 0.524s 2022-11-23T02:32:19.7008497Z 2022-11-23T02:32:19.7008566Z OK (skipped=1) 2022-11-23T02:32:19.7008570Z 2022-11-23T02:32:19.7008646Z Generating XML reports... 2022-11-23T02:32:19.7008990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022735.xml 2022-11-23T02:32:19.7009221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7009499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7009616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7009942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7010071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7010076Z 2022-11-23T02:32:19.7010141Z Running tests... 2022-11-23T02:32:19.7010337Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7011049Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.521s) 2022-11-23T02:32:19.7011055Z 2022-11-23T02:32:19.7011246Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7011312Z Ran 1 test in 0.521s 2022-11-23T02:32:19.7011319Z 2022-11-23T02:32:19.7011382Z OK (skipped=1) 2022-11-23T02:32:19.7011386Z 2022-11-23T02:32:19.7011463Z Generating XML reports... 2022-11-23T02:32:19.7011802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022739.xml 2022-11-23T02:32:19.7012067Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7012348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7012464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7012747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7012875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7012881Z 2022-11-23T02:32:19.7012941Z Running tests... 2022-11-23T02:32:19.7013139Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7013370Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16735 2022-11-23T02:32:19.7013521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16736 2022-11-23T02:32:19.7013711Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7013988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7014105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7014388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7014515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7014687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7014962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7015079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7015362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7015488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7015661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7015964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7016262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7016453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7016611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7016780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.7016950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.7017246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7017539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7017790Z STAGE:2022-11-23 02:27:45 16735:16735 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7018034Z STAGE:2022-11-23 02:27:45 16736:16736 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7018290Z STAGE:2022-11-23 02:27:45 16736:16736 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7018771Z STAGE:2022-11-23 02:27:45 16735:16735 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:27:45 16736:16736 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7018778Z 2022-11-23T02:32:19.7019049Z STAGE:2022-11-23 02:27:45 16735:16735 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7019299Z STAGE:2022-11-23 02:27:45 16736:16736 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7019542Z STAGE:2022-11-23 02:27:45 16735:16735 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7019793Z STAGE:2022-11-23 02:27:45 16735:16735 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7020053Z STAGE:2022-11-23 02:27:45 16735:16735 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7020307Z STAGE:2022-11-23 02:27:45 16736:16736 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7020564Z STAGE:2022-11-23 02:27:45 16736:16736 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7020624Z ok (4.039s) 2022-11-23T02:32:19.7020632Z 2022-11-23T02:32:19.7020829Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7020897Z Ran 1 test in 4.039s 2022-11-23T02:32:19.7020903Z 2022-11-23T02:32:19.7020955Z OK 2022-11-23T02:32:19.7020959Z 2022-11-23T02:32:19.7021036Z Generating XML reports... 2022-11-23T02:32:19.7021377Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022742.xml 2022-11-23T02:32:19.7021610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7021887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7022005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7022289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7022420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7022425Z 2022-11-23T02:32:19.7022490Z Running tests... 2022-11-23T02:32:19.7022688Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7022918Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16930 2022-11-23T02:32:19.7023064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16931 2022-11-23T02:32:19.7023253Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7023565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7023682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7023969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7024098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7024268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7024543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7024660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7024941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7025071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7025243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7025574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7025875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7026031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7026189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7026357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.7026524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.7026820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7027111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7027369Z STAGE:2022-11-23 02:27:52 16930:16930 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7027614Z STAGE:2022-11-23 02:27:52 16931:16931 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7027866Z STAGE:2022-11-23 02:27:52 16931:16931 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7028294Z STAGE:2022-11-23 02:27:52 16930:16930 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:27:52 16931:16931 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7028299Z 2022-11-23T02:32:19.7028562Z STAGE:2022-11-23 02:27:52 16930:16930 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7028813Z STAGE:2022-11-23 02:27:52 16931:16931 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7029051Z STAGE:2022-11-23 02:27:52 16930:16930 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7029306Z STAGE:2022-11-23 02:27:52 16930:16930 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7029566Z STAGE:2022-11-23 02:27:52 16930:16930 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7029817Z STAGE:2022-11-23 02:27:52 16931:16931 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7030077Z STAGE:2022-11-23 02:27:52 16931:16931 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7030137Z ok (4.042s) 2022-11-23T02:32:19.7030142Z 2022-11-23T02:32:19.7030338Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7030440Z Ran 1 test in 4.042s 2022-11-23T02:32:19.7030445Z 2022-11-23T02:32:19.7030498Z OK 2022-11-23T02:32:19.7030502Z 2022-11-23T02:32:19.7030581Z Generating XML reports... 2022-11-23T02:32:19.7030931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022749.xml 2022-11-23T02:32:19.7031166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7031441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7031559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7031843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7031972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7031977Z 2022-11-23T02:32:19.7032044Z Running tests... 2022-11-23T02:32:19.7032238Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7032474Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17125 2022-11-23T02:32:19.7032658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17126 2022-11-23T02:32:19.7032853Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7033130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7033248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7033527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7033656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7033831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7034130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7034408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7034524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7034808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7034936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7035105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7035405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7035566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7035723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7035894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.7036114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.7036415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7036711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7036966Z STAGE:2022-11-23 02:27:59 17125:17125 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7037254Z STAGE:2022-11-23 02:27:59 17126:17126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7037509Z STAGE:2022-11-23 02:27:59 17126:17126 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7037775Z STAGE:2022-11-23 02:27:59 17126:17126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7038028Z STAGE:2022-11-23 02:27:59 17125:17125 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7038289Z STAGE:2022-11-23 02:27:59 17125:17125 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7038533Z STAGE:2022-11-23 02:27:59 17126:17126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7038774Z STAGE:2022-11-23 02:27:59 17125:17125 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7039017Z STAGE:2022-11-23 02:27:59 17125:17125 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7039266Z STAGE:2022-11-23 02:27:59 17126:17126 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7039526Z STAGE:2022-11-23 02:27:59 17125:17125 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7039817Z STAGE:2022-11-23 02:27:59 17126:17126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7039880Z ok (4.141s) 2022-11-23T02:32:19.7039886Z 2022-11-23T02:32:19.7040083Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7040151Z Ran 1 test in 4.142s 2022-11-23T02:32:19.7040156Z 2022-11-23T02:32:19.7040207Z OK 2022-11-23T02:32:19.7040212Z 2022-11-23T02:32:19.7040288Z Generating XML reports... 2022-11-23T02:32:19.7040629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022756.xml 2022-11-23T02:32:19.7040863Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7041144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7041261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7041546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7041675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7041679Z 2022-11-23T02:32:19.7041745Z Running tests... 2022-11-23T02:32:19.7041939Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7042170Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17320 2022-11-23T02:32:19.7042321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17321 2022-11-23T02:32:19.7042510Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7042788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7042904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7043186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7043318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7043489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7043765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7043880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7044162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7044325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7044495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7044801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7045100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7045256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7045413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7045580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.7045881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7046049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.7046395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7046653Z STAGE:2022-11-23 02:28:06 17320:17320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7046900Z STAGE:2022-11-23 02:28:06 17321:17321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7047155Z STAGE:2022-11-23 02:28:06 17321:17321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7047402Z STAGE:2022-11-23 02:28:06 17320:17320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7047663Z STAGE:2022-11-23 02:28:06 17321:17321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7047925Z STAGE:2022-11-23 02:28:06 17320:17320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7048167Z STAGE:2022-11-23 02:28:06 17321:17321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7048410Z STAGE:2022-11-23 02:28:06 17320:17320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7048652Z STAGE:2022-11-23 02:28:06 17320:17320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7048910Z STAGE:2022-11-23 02:28:06 17320:17320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7049162Z STAGE:2022-11-23 02:28:06 17321:17321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7049420Z STAGE:2022-11-23 02:28:06 17321:17321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7049481Z ok (3.838s) 2022-11-23T02:32:19.7049489Z 2022-11-23T02:32:19.7049689Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7049759Z Ran 1 test in 3.839s 2022-11-23T02:32:19.7049764Z 2022-11-23T02:32:19.7049817Z OK 2022-11-23T02:32:19.7049822Z 2022-11-23T02:32:19.7049899Z Generating XML reports... 2022-11-23T02:32:19.7050245Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022804.xml 2022-11-23T02:32:19.7050481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7050756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7050873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7051158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7051288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7051329Z 2022-11-23T02:32:19.7051394Z Running tests... 2022-11-23T02:32:19.7051593Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7051822Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17515 2022-11-23T02:32:19.7051974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17516 2022-11-23T02:32:19.7052168Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7052611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7052736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7053032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7053170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7053352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7053681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7053805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7054107Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7054241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7054421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7054739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7055052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7055219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7055387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7055497Z skip: Skipped due to small world size. (3.839s) 2022-11-23T02:32:19.7055503Z 2022-11-23T02:32:19.7055709Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7055780Z Ran 1 test in 3.839s 2022-11-23T02:32:19.7055785Z 2022-11-23T02:32:19.7055851Z OK (skipped=1) 2022-11-23T02:32:19.7055856Z 2022-11-23T02:32:19.7055938Z Generating XML reports... 2022-11-23T02:32:19.7056298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022811.xml 2022-11-23T02:32:19.7056544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7056836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7056959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7057262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7057398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7057403Z 2022-11-23T02:32:19.7057467Z Running tests... 2022-11-23T02:32:19.7057674Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7057908Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17698 2022-11-23T02:32:19.7058071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17699 2022-11-23T02:32:19.7058304Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7058594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7058717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7059015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7059148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7059329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7059646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7059936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7060061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7060359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7060495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7060711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7061031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7061198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7061364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7061474Z skip: Skipped due to small world size. (3.740s) 2022-11-23T02:32:19.7061480Z 2022-11-23T02:32:19.7061685Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7061763Z Ran 1 test in 3.741s 2022-11-23T02:32:19.7061768Z 2022-11-23T02:32:19.7061834Z OK (skipped=1) 2022-11-23T02:32:19.7061840Z 2022-11-23T02:32:19.7061914Z Generating XML reports... 2022-11-23T02:32:19.7062276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022817.xml 2022-11-23T02:32:19.7062520Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7062808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7062931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7063228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7063365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7063372Z 2022-11-23T02:32:19.7063441Z Running tests... 2022-11-23T02:32:19.7063645Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7063889Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17881 2022-11-23T02:32:19.7064153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17882 2022-11-23T02:32:19.7064344Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7064621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7064738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7065021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7065184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7065357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7065660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7065938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7066054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7066337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7066465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7066630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7066931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7067090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7067246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7067382Z skip: Skipped due to small world size. (3.940s) 2022-11-23T02:32:19.7067387Z 2022-11-23T02:32:19.7067588Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7067656Z Ran 1 test in 3.941s 2022-11-23T02:32:19.7067661Z 2022-11-23T02:32:19.7067725Z OK (skipped=1) 2022-11-23T02:32:19.7067729Z 2022-11-23T02:32:19.7067805Z Generating XML reports... 2022-11-23T02:32:19.7068149Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022824.xml 2022-11-23T02:32:19.7068381Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7068660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7068778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7069062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7069191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7069196Z 2022-11-23T02:32:19.7069261Z Running tests... 2022-11-23T02:32:19.7069455Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7069680Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18064 2022-11-23T02:32:19.7069830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18065 2022-11-23T02:32:19.7070021Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7070297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7070414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7070696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7070822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7070993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7071268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7071383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7071666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7071827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7071997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7072302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7072602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7072760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7072915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7073020Z skip: Skipped due to small world size. (3.842s) 2022-11-23T02:32:19.7073026Z 2022-11-23T02:32:19.7073221Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7073290Z Ran 1 test in 3.843s 2022-11-23T02:32:19.7073296Z 2022-11-23T02:32:19.7073359Z OK (skipped=1) 2022-11-23T02:32:19.7073363Z 2022-11-23T02:32:19.7073440Z Generating XML reports... 2022-11-23T02:32:19.7073812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022831.xml 2022-11-23T02:32:19.7074046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7074324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7074440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7074723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7074852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7074858Z 2022-11-23T02:32:19.7074923Z Running tests... 2022-11-23T02:32:19.7075111Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7075336Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18247 2022-11-23T02:32:19.7075490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18248 2022-11-23T02:32:19.7075680Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7075954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7076114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7076567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7076702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7076886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7077180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7081091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7081442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7081587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7081766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7082096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7082417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7082679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7082962Z STAGE:2022-11-23 02:28:40 18248:18248 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7083140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7083409Z STAGE:2022-11-23 02:28:41 18247:18247 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7083862Z STAGE:2022-11-23 02:28:41 18248:18248 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:28:41 18247:18247 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7083869Z 2022-11-23T02:32:19.7084157Z STAGE:2022-11-23 02:28:41 18248:18248 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7084440Z STAGE:2022-11-23 02:28:41 18247:18247 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7084704Z STAGE:2022-11-23 02:28:41 18248:18248 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7084956Z STAGE:2022-11-23 02:28:41 18247:18247 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7085260Z STAGE:2022-11-23 02:28:41 18247:18247 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7085537Z STAGE:2022-11-23 02:28:41 18247:18247 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7085800Z STAGE:2022-11-23 02:28:41 18248:18248 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7086071Z STAGE:2022-11-23 02:28:41 18248:18248 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7086135Z ok (3.839s) 2022-11-23T02:32:19.7086141Z 2022-11-23T02:32:19.7086350Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7086429Z Ran 1 test in 3.840s 2022-11-23T02:32:19.7086434Z 2022-11-23T02:32:19.7086490Z OK 2022-11-23T02:32:19.7086495Z 2022-11-23T02:32:19.7086576Z Generating XML reports... 2022-11-23T02:32:19.7087048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022838.xml 2022-11-23T02:32:19.7087288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7087569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7087686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7087965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7088098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7088103Z 2022-11-23T02:32:19.7088171Z Running tests... 2022-11-23T02:32:19.7088370Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7088592Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18436 2022-11-23T02:32:19.7088746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18437 2022-11-23T02:32:19.7088938Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7089216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7089337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7089624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7089752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7089963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7090267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7090545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7090663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7090946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7091073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7091244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7091547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7091708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7091964Z STAGE:2022-11-23 02:28:47 18437:18437 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7092153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7092408Z STAGE:2022-11-23 02:28:47 18436:18436 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7092656Z STAGE:2022-11-23 02:28:47 18436:18436 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7092902Z STAGE:2022-11-23 02:28:47 18437:18437 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7093166Z STAGE:2022-11-23 02:28:47 18436:18436 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7093426Z STAGE:2022-11-23 02:28:47 18437:18437 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7093673Z STAGE:2022-11-23 02:28:47 18437:18437 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7093912Z STAGE:2022-11-23 02:28:47 18436:18436 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7094161Z STAGE:2022-11-23 02:28:47 18436:18436 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7094590Z STAGE:2022-11-23 02:28:47 18437:18437 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:28:47 18436:18436 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7094595Z 2022-11-23T02:32:19.7094857Z STAGE:2022-11-23 02:28:47 18437:18437 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7094917Z ok (3.928s) 2022-11-23T02:32:19.7094922Z 2022-11-23T02:32:19.7095119Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7095190Z Ran 1 test in 3.929s 2022-11-23T02:32:19.7095195Z 2022-11-23T02:32:19.7095248Z OK 2022-11-23T02:32:19.7095252Z 2022-11-23T02:32:19.7095328Z Generating XML reports... 2022-11-23T02:32:19.7095672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022845.xml 2022-11-23T02:32:19.7095908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7096184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7096303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7096585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7096714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7096719Z 2022-11-23T02:32:19.7096783Z Running tests... 2022-11-23T02:32:19.7097016Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7097213Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s) 2022-11-23T02:32:19.7097218Z 2022-11-23T02:32:19.7097415Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7097484Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7097489Z 2022-11-23T02:32:19.7097552Z OK (skipped=1) 2022-11-23T02:32:19.7097557Z 2022-11-23T02:32:19.7097633Z Generating XML reports... 2022-11-23T02:32:19.7097975Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022852.xml 2022-11-23T02:32:19.7098210Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7098489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7098608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7098892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7099021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7099026Z 2022-11-23T02:32:19.7099123Z Running tests... 2022-11-23T02:32:19.7099315Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7099542Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18683 2022-11-23T02:32:19.7099695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18684 2022-11-23T02:32:19.7099884Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7100157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7100275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7100557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7100686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7100856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7101131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7101246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7101531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7101659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7101831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7102133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7102433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7102593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7102848Z STAGE:2022-11-23 02:28:57 18684:18684 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7103008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7103258Z STAGE:2022-11-23 02:28:57 18683:18683 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7103681Z STAGE:2022-11-23 02:28:57 18683:18683 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:28:57 18684:18684 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7103734Z 2022-11-23T02:32:19.7104001Z STAGE:2022-11-23 02:28:57 18683:18683 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7104266Z STAGE:2022-11-23 02:28:57 18684:18684 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7104514Z STAGE:2022-11-23 02:28:57 18684:18684 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7104754Z STAGE:2022-11-23 02:28:57 18683:18683 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7105004Z STAGE:2022-11-23 02:28:57 18684:18684 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7105244Z STAGE:2022-11-23 02:28:57 18683:18683 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7105503Z STAGE:2022-11-23 02:28:57 18684:18684 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7105763Z STAGE:2022-11-23 02:28:57 18683:18683 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7105823Z ok (3.938s) 2022-11-23T02:32:19.7105828Z 2022-11-23T02:32:19.7106060Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7106131Z Ran 1 test in 3.939s 2022-11-23T02:32:19.7106137Z 2022-11-23T02:32:19.7106190Z OK 2022-11-23T02:32:19.7106194Z 2022-11-23T02:32:19.7106271Z Generating XML reports... 2022-11-23T02:32:19.7106616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022855.xml 2022-11-23T02:32:19.7106852Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7107131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7107252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7107533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7107661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7107666Z 2022-11-23T02:32:19.7107732Z Running tests... 2022-11-23T02:32:19.7107927Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7108138Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s) 2022-11-23T02:32:19.7108143Z 2022-11-23T02:32:19.7108335Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7108401Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7108406Z 2022-11-23T02:32:19.7108469Z OK (skipped=1) 2022-11-23T02:32:19.7108473Z 2022-11-23T02:32:19.7108549Z Generating XML reports... 2022-11-23T02:32:19.7108892Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022902.xml 2022-11-23T02:32:19.7109123Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7109398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7109515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7109798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7109925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7109930Z 2022-11-23T02:32:19.7109994Z Running tests... 2022-11-23T02:32:19.7110186Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7110381Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.002s) 2022-11-23T02:32:19.7110418Z 2022-11-23T02:32:19.7110613Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7110681Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7110686Z 2022-11-23T02:32:19.7110749Z OK (skipped=1) 2022-11-23T02:32:19.7110754Z 2022-11-23T02:32:19.7110830Z Generating XML reports... 2022-11-23T02:32:19.7111171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022905.xml 2022-11-23T02:32:19.7111402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7111676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7111792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7112074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7112205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7112209Z 2022-11-23T02:32:19.7112275Z Running tests... 2022-11-23T02:32:19.7112467Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7112721Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18988 2022-11-23T02:32:19.7112873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18989 2022-11-23T02:32:19.7113063Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7113342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7113457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7113739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7113872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7114045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7114320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7114437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7114720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7114846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7115018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7115324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7115627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7115786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7116043Z STAGE:2022-11-23 02:29:10 18989:18989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7116252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7116507Z STAGE:2022-11-23 02:29:10 18988:18988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7116761Z STAGE:2022-11-23 02:29:10 18988:18988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7117009Z STAGE:2022-11-23 02:29:10 18989:18989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7117270Z STAGE:2022-11-23 02:29:10 18988:18988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7117574Z STAGE:2022-11-23 02:29:10 18989:18989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7117819Z STAGE:2022-11-23 02:29:10 18989:18989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7118063Z STAGE:2022-11-23 02:29:10 18988:18988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7118312Z STAGE:2022-11-23 02:29:10 18988:18988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7118568Z STAGE:2022-11-23 02:29:10 18988:18988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7118813Z STAGE:2022-11-23 02:29:10 18989:18989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7119069Z STAGE:2022-11-23 02:29:10 18989:18989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7119130Z ok (3.829s) 2022-11-23T02:32:19.7119136Z 2022-11-23T02:32:19.7119332Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7119399Z Ran 1 test in 3.830s 2022-11-23T02:32:19.7119404Z 2022-11-23T02:32:19.7119456Z OK 2022-11-23T02:32:19.7119461Z 2022-11-23T02:32:19.7119537Z Generating XML reports... 2022-11-23T02:32:19.7119916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022908.xml 2022-11-23T02:32:19.7120151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7120433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7120552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7120836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7120967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7120972Z 2022-11-23T02:32:19.7121038Z Running tests... 2022-11-23T02:32:19.7121234Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7121420Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-11-23T02:32:19.7121425Z 2022-11-23T02:32:19.7121617Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7121684Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7121688Z 2022-11-23T02:32:19.7121752Z OK (skipped=1) 2022-11-23T02:32:19.7121756Z 2022-11-23T02:32:19.7121833Z Generating XML reports... 2022-11-23T02:32:19.7122175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022915.xml 2022-11-23T02:32:19.7122407Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7122683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7122801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7123085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7123213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7123217Z 2022-11-23T02:32:19.7123281Z Running tests... 2022-11-23T02:32:19.7123475Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7123663Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-11-23T02:32:19.7123668Z 2022-11-23T02:32:19.7123859Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7123927Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7123932Z 2022-11-23T02:32:19.7124030Z OK (skipped=1) 2022-11-23T02:32:19.7124035Z 2022-11-23T02:32:19.7124111Z Generating XML reports... 2022-11-23T02:32:19.7124453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022918.xml 2022-11-23T02:32:19.7124688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7124966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7125085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7125367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7125497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7125502Z 2022-11-23T02:32:19.7125566Z Running tests... 2022-11-23T02:32:19.7125759Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7125985Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19293 2022-11-23T02:32:19.7126138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19294 2022-11-23T02:32:19.7126359Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7126637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7126750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7127032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7127161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7127333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7127636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7127910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7128028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7128313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7128442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7128609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7128914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7129073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7129331Z STAGE:2022-11-23 02:29:23 19294:19294 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7129486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7129741Z STAGE:2022-11-23 02:29:23 19293:19293 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7129996Z STAGE:2022-11-23 02:29:23 19293:19293 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7130433Z STAGE:2022-11-23 02:29:23 19294:19294 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:29:23 19293:19293 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7130438Z 2022-11-23T02:32:19.7130699Z STAGE:2022-11-23 02:29:23 19294:19294 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7130947Z STAGE:2022-11-23 02:29:23 19294:19294 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7131222Z STAGE:2022-11-23 02:29:23 19293:19293 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7131474Z STAGE:2022-11-23 02:29:23 19293:19293 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7131736Z STAGE:2022-11-23 02:29:23 19293:19293 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7131985Z STAGE:2022-11-23 02:29:23 19294:19294 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7132244Z STAGE:2022-11-23 02:29:23 19294:19294 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7132303Z ok (3.847s) 2022-11-23T02:32:19.7132308Z 2022-11-23T02:32:19.7132501Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7132570Z Ran 1 test in 3.848s 2022-11-23T02:32:19.7132575Z 2022-11-23T02:32:19.7132630Z OK 2022-11-23T02:32:19.7132635Z 2022-11-23T02:32:19.7132712Z Generating XML reports... 2022-11-23T02:32:19.7133054Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022920.xml 2022-11-23T02:32:19.7133288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7133597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7133714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7133998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7134126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7134132Z 2022-11-23T02:32:19.7134197Z Running tests... 2022-11-23T02:32:19.7134391Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7134614Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19482 2022-11-23T02:32:19.7134764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19483 2022-11-23T02:32:19.7134954Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7135227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7135344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7135628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7135755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7135926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7136230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7136507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7136621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7136906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7137034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7137202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7137502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7137660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7137949Z STAGE:2022-11-23 02:29:30 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7138105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7138360Z STAGE:2022-11-23 02:29:30 19482:19482 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7138612Z STAGE:2022-11-23 02:29:30 19482:19482 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7138874Z STAGE:2022-11-23 02:29:30 19482:19482 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7139125Z STAGE:2022-11-23 02:29:30 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7139384Z STAGE:2022-11-23 02:29:30 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7139628Z STAGE:2022-11-23 02:29:30 19482:19482 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7139869Z STAGE:2022-11-23 02:29:30 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7140117Z STAGE:2022-11-23 02:29:30 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7140393Z STAGE:2022-11-23 02:29:30 19482:19482 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7140654Z STAGE:2022-11-23 02:29:30 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7140912Z STAGE:2022-11-23 02:29:30 19482:19482 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7140971Z ok (3.841s) 2022-11-23T02:32:19.7140976Z 2022-11-23T02:32:19.7141173Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7141242Z Ran 1 test in 3.842s 2022-11-23T02:32:19.7141247Z 2022-11-23T02:32:19.7141300Z OK 2022-11-23T02:32:19.7141307Z 2022-11-23T02:32:19.7141379Z Generating XML reports... 2022-11-23T02:32:19.7141720Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022927.xml 2022-11-23T02:32:19.7141953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7142232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7142350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7142634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7142763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7142768Z 2022-11-23T02:32:19.7142832Z Running tests... 2022-11-23T02:32:19.7143028Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7143254Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19671 2022-11-23T02:32:19.7143406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19672 2022-11-23T02:32:19.7143597Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7143867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7143983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7144266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7144394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7144564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7144879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7144994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7145280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7145408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7145580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7145877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7146173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7146333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7146491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7146551Z ok (3.643s) 2022-11-23T02:32:19.7146556Z 2022-11-23T02:32:19.7146751Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7146819Z Ran 1 test in 3.644s 2022-11-23T02:32:19.7146855Z 2022-11-23T02:32:19.7146910Z OK 2022-11-23T02:32:19.7146914Z 2022-11-23T02:32:19.7146990Z Generating XML reports... 2022-11-23T02:32:19.7147332Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022934.xml 2022-11-23T02:32:19.7147564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7147837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7147956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7148238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7148365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7148370Z 2022-11-23T02:32:19.7148434Z Running tests... 2022-11-23T02:32:19.7148633Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7148862Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19854 2022-11-23T02:32:19.7149013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19855 2022-11-23T02:32:19.7149199Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7149470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7149589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7149869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7149991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7150164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7150440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7150557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7150839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7150966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7151137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7151472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7151768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7151927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7152182Z STAGE:2022-11-23 02:29:43 19855:19855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7152339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7152591Z STAGE:2022-11-23 02:29:43 19854:19854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7152842Z STAGE:2022-11-23 02:29:43 19855:19855 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7153087Z STAGE:2022-11-23 02:29:43 19854:19854 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7153349Z STAGE:2022-11-23 02:29:43 19855:19855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7153595Z STAGE:2022-11-23 02:29:43 19855:19855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7153905Z STAGE:2022-11-23 02:29:43 19854:19854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7154151Z STAGE:2022-11-23 02:29:43 19854:19854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7154398Z STAGE:2022-11-23 02:29:43 19855:19855 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7154656Z STAGE:2022-11-23 02:29:43 19855:19855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7154903Z STAGE:2022-11-23 02:29:43 19854:19854 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7155162Z STAGE:2022-11-23 02:29:43 19854:19854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7155217Z ok (3.942s) 2022-11-23T02:32:19.7155226Z 2022-11-23T02:32:19.7155417Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7155485Z Ran 1 test in 3.942s 2022-11-23T02:32:19.7155490Z 2022-11-23T02:32:19.7155548Z OK 2022-11-23T02:32:19.7155553Z 2022-11-23T02:32:19.7155629Z Generating XML reports... 2022-11-23T02:32:19.7155972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022941.xml 2022-11-23T02:32:19.7156254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7156530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7156647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7156932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7157059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7157064Z 2022-11-23T02:32:19.7157129Z Running tests... 2022-11-23T02:32:19.7157329Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7157509Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s) 2022-11-23T02:32:19.7157514Z 2022-11-23T02:32:19.7157710Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7157777Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7157782Z 2022-11-23T02:32:19.7157846Z OK (skipped=1) 2022-11-23T02:32:19.7157851Z 2022-11-23T02:32:19.7157928Z Generating XML reports... 2022-11-23T02:32:19.7158267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022948.xml 2022-11-23T02:32:19.7158580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7158853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7158971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7159253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7159382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7159387Z 2022-11-23T02:32:19.7159452Z Running tests... 2022-11-23T02:32:19.7159647Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7159838Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-11-23T02:32:19.7159843Z 2022-11-23T02:32:19.7160033Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7160103Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7160107Z 2022-11-23T02:32:19.7160171Z OK (skipped=1) 2022-11-23T02:32:19.7160176Z 2022-11-23T02:32:19.7160252Z Generating XML reports... 2022-11-23T02:32:19.7160627Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022951.xml 2022-11-23T02:32:19.7160861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7161134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7161250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7161532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7161660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7161668Z 2022-11-23T02:32:19.7161732Z Running tests... 2022-11-23T02:32:19.7161925Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7162151Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20159 2022-11-23T02:32:19.7162302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20160 2022-11-23T02:32:19.7162494Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7162768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7162885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7163168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7163293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7163464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7163739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7163857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7164136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7164266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7164436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7164738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7165037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7165293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7165449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7165619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:32:19.7165787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:32:19.7166086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7166381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:32:19.7166635Z STAGE:2022-11-23 02:29:56 20160:20160 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7166882Z STAGE:2022-11-23 02:29:56 20159:20159 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7167139Z STAGE:2022-11-23 02:29:56 20159:20159 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7167605Z STAGE:2022-11-23 02:29:56 20160:20160 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:29:56 20159:20159 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7167611Z 2022-11-23T02:32:19.7167875Z STAGE:2022-11-23 02:29:56 20160:20160 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7168123Z STAGE:2022-11-23 02:29:56 20159:20159 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7168366Z STAGE:2022-11-23 02:29:56 20160:20160 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7168615Z STAGE:2022-11-23 02:29:56 20160:20160 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7169048Z STAGE:2022-11-23 02:29:56 20160:20160 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:29:56 20159:20159 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7169053Z 2022-11-23T02:32:19.7169317Z STAGE:2022-11-23 02:29:56 20159:20159 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7169377Z ok (3.944s) 2022-11-23T02:32:19.7169382Z 2022-11-23T02:32:19.7169580Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7169648Z Ran 1 test in 3.944s 2022-11-23T02:32:19.7169653Z 2022-11-23T02:32:19.7169701Z OK 2022-11-23T02:32:19.7169710Z 2022-11-23T02:32:19.7169781Z Generating XML reports... 2022-11-23T02:32:19.7170123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022954.xml 2022-11-23T02:32:19.7170354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7170634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7170751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7171038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7171166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7171171Z 2022-11-23T02:32:19.7171235Z Running tests... 2022-11-23T02:32:19.7171429Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7171654Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20354 2022-11-23T02:32:19.7171808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20355 2022-11-23T02:32:19.7172032Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7172308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7172425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7172711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7172840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7173011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7173290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7173407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7173690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7173822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7173990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7174320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7174620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7174778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7174935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7175041Z skip: Skipped due to small world size. (3.737s) 2022-11-23T02:32:19.7175046Z 2022-11-23T02:32:19.7175243Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7175313Z Ran 1 test in 3.738s 2022-11-23T02:32:19.7175318Z 2022-11-23T02:32:19.7175382Z OK (skipped=1) 2022-11-23T02:32:19.7175386Z 2022-11-23T02:32:19.7175464Z Generating XML reports... 2022-11-23T02:32:19.7175806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023001.xml 2022-11-23T02:32:19.7176038Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7176314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7176429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7176715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7176842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7176849Z 2022-11-23T02:32:19.7176915Z Running tests... 2022-11-23T02:32:19.7177110Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7177340Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20537 2022-11-23T02:32:19.7177491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20538 2022-11-23T02:32:19.7177684Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7177960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7178077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7178356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7178519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7178691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7178965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7179082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7179366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7179493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7179662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7179964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7180263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7180424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7180582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7180673Z ok (3.941s) 2022-11-23T02:32:19.7180678Z 2022-11-23T02:32:19.7180875Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7180943Z Ran 1 test in 3.941s 2022-11-23T02:32:19.7180947Z 2022-11-23T02:32:19.7181000Z OK 2022-11-23T02:32:19.7181004Z 2022-11-23T02:32:19.7181079Z Generating XML reports... 2022-11-23T02:32:19.7181420Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023007.xml 2022-11-23T02:32:19.7181654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7181930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7182048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7182332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7182459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7182467Z 2022-11-23T02:32:19.7182528Z Running tests... 2022-11-23T02:32:19.7182726Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7182945Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20720 2022-11-23T02:32:19.7183096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20721 2022-11-23T02:32:19.7183285Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7183561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7183678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7183963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7184091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7184261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7184537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7184653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7184937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7185100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7185270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7185573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7185876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7186036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7186193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7186252Z ok (3.833s) 2022-11-23T02:32:19.7186258Z 2022-11-23T02:32:19.7186455Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7186518Z Ran 1 test in 3.833s 2022-11-23T02:32:19.7186529Z 2022-11-23T02:32:19.7186577Z OK 2022-11-23T02:32:19.7186581Z 2022-11-23T02:32:19.7186657Z Generating XML reports... 2022-11-23T02:32:19.7186999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023014.xml 2022-11-23T02:32:19.7187265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7187543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7187660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7187944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7188072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7188077Z 2022-11-23T02:32:19.7188141Z Running tests... 2022-11-23T02:32:19.7188338Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7188574Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20903 2022-11-23T02:32:19.7188723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20904 2022-11-23T02:32:19.7188916Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7189192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7189308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7189590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7189718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7189888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7190165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7190281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7190564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7190690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7190858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7191161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7191460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7191653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7191811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7191870Z ok (3.839s) 2022-11-23T02:32:19.7191875Z 2022-11-23T02:32:19.7192075Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7192143Z Ran 1 test in 3.839s 2022-11-23T02:32:19.7192148Z 2022-11-23T02:32:19.7192200Z OK 2022-11-23T02:32:19.7192204Z 2022-11-23T02:32:19.7192279Z Generating XML reports... 2022-11-23T02:32:19.7192620Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023021.xml 2022-11-23T02:32:19.7192854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7193132Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7193250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7193531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7193657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7193662Z 2022-11-23T02:32:19.7193758Z Running tests... 2022-11-23T02:32:19.7193957Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7194206Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21086 2022-11-23T02:32:19.7194358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21087 2022-11-23T02:32:19.7194551Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7194827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7194942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7195224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7195352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7195522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7195824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7196170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7196287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7196571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7196702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7196871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7197174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7197331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7197589Z STAGE:2022-11-23 02:30:31 21087:21087 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7197750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7198002Z STAGE:2022-11-23 02:30:31 21086:21086 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7198255Z STAGE:2022-11-23 02:30:31 21086:21086 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7198561Z STAGE:2022-11-23 02:30:31 21086:21086 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7198811Z STAGE:2022-11-23 02:30:31 21087:21087 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7199074Z STAGE:2022-11-23 02:30:31 21087:21087 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7199134Z ok (4.058s) 2022-11-23T02:32:19.7199139Z 2022-11-23T02:32:19.7199340Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7199408Z Ran 1 test in 4.058s 2022-11-23T02:32:19.7199414Z 2022-11-23T02:32:19.7199467Z OK 2022-11-23T02:32:19.7199471Z 2022-11-23T02:32:19.7199543Z Generating XML reports... 2022-11-23T02:32:19.7199884Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023028.xml 2022-11-23T02:32:19.7200120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7200402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7200518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7200837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7200966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7200972Z 2022-11-23T02:32:19.7201037Z Running tests... 2022-11-23T02:32:19.7201235Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7201479Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21275 2022-11-23T02:32:19.7201631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21276 2022-11-23T02:32:19.7201825Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7202100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7202219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7202501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7202629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7202802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7203105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7203381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7203501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7203783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7203912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7204082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7204378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7204538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7204793Z STAGE:2022-11-23 02:30:38 21276:21276 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7204951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7205253Z STAGE:2022-11-23 02:30:38 21275:21275 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7205506Z STAGE:2022-11-23 02:30:38 21275:21275 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7205771Z STAGE:2022-11-23 02:30:38 21275:21275 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7206022Z STAGE:2022-11-23 02:30:38 21276:21276 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7206284Z STAGE:2022-11-23 02:30:38 21276:21276 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7206344Z ok (3.842s) 2022-11-23T02:32:19.7206349Z 2022-11-23T02:32:19.7206544Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7206611Z Ran 1 test in 3.843s 2022-11-23T02:32:19.7206616Z 2022-11-23T02:32:19.7206668Z OK 2022-11-23T02:32:19.7206672Z 2022-11-23T02:32:19.7206751Z Generating XML reports... 2022-11-23T02:32:19.7207094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023035.xml 2022-11-23T02:32:19.7207332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7207642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7207762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7208049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7208180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7208184Z 2022-11-23T02:32:19.7208249Z Running tests... 2022-11-23T02:32:19.7208446Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7208681Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21464 2022-11-23T02:32:19.7208836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21465 2022-11-23T02:32:19.7209029Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7209304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7209420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7209702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7209830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7209999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7210303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7210580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7210697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7210982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7211109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7211281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7211580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7211739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7212025Z STAGE:2022-11-23 02:30:45 21465:21465 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7212182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7212438Z STAGE:2022-11-23 02:30:45 21464:21464 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7212689Z STAGE:2022-11-23 02:30:45 21465:21465 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7212951Z STAGE:2022-11-23 02:30:45 21465:21465 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7213203Z STAGE:2022-11-23 02:30:45 21464:21464 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7213463Z STAGE:2022-11-23 02:30:45 21464:21464 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7213519Z ok (3.957s) 2022-11-23T02:32:19.7213528Z 2022-11-23T02:32:19.7213719Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7213789Z Ran 1 test in 3.958s 2022-11-23T02:32:19.7213794Z 2022-11-23T02:32:19.7213848Z OK 2022-11-23T02:32:19.7213852Z 2022-11-23T02:32:19.7213927Z Generating XML reports... 2022-11-23T02:32:19.7214302Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023042.xml 2022-11-23T02:32:19.7214536Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7214810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7214926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7215208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7215334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7215341Z 2022-11-23T02:32:19.7215406Z Running tests... 2022-11-23T02:32:19.7215600Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7215766Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s) 2022-11-23T02:32:19.7215771Z 2022-11-23T02:32:19.7215966Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7216033Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7216037Z 2022-11-23T02:32:19.7216100Z OK (skipped=1) 2022-11-23T02:32:19.7216105Z 2022-11-23T02:32:19.7216181Z Generating XML reports... 2022-11-23T02:32:19.7216519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023049.xml 2022-11-23T02:32:19.7216750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7217021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7217140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7217418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7217544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7217551Z 2022-11-23T02:32:19.7217616Z Running tests... 2022-11-23T02:32:19.7217808Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7217993Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s) 2022-11-23T02:32:19.7217998Z 2022-11-23T02:32:19.7218190Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7218258Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7218262Z 2022-11-23T02:32:19.7218325Z OK (skipped=1) 2022-11-23T02:32:19.7218329Z 2022-11-23T02:32:19.7218406Z Generating XML reports... 2022-11-23T02:32:19.7218785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023052.xml 2022-11-23T02:32:19.7219018Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7219297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7219414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7219698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7219826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7219830Z 2022-11-23T02:32:19.7219895Z Running tests... 2022-11-23T02:32:19.7220088Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7220272Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-11-23T02:32:19.7220279Z 2022-11-23T02:32:19.7220468Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7220535Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7220540Z 2022-11-23T02:32:19.7220602Z OK (skipped=1) 2022-11-23T02:32:19.7220606Z 2022-11-23T02:32:19.7220716Z Generating XML reports... 2022-11-23T02:32:19.7221057Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023055.xml 2022-11-23T02:32:19.7221284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7221561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7221677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7221959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7222089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7222094Z 2022-11-23T02:32:19.7222158Z Running tests... 2022-11-23T02:32:19.7222350Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7222585Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21827 2022-11-23T02:32:19.7222738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21828 2022-11-23T02:32:19.7222927Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7223202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7223319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7223604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7223731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7223902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7224206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7224482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7224599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7224882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7225009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7225210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7225512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7225671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7225926Z STAGE:2022-11-23 02:31:01 21828:21828 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7226087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7226338Z STAGE:2022-11-23 02:31:01 21827:21827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7226594Z STAGE:2022-11-23 02:31:01 21828:21828 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7226855Z STAGE:2022-11-23 02:31:01 21828:21828 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7227111Z STAGE:2022-11-23 02:31:01 21827:21827 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7227372Z STAGE:2022-11-23 02:31:01 21827:21827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7227433Z ok (3.829s) 2022-11-23T02:32:19.7227438Z 2022-11-23T02:32:19.7227665Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7227735Z Ran 1 test in 3.830s 2022-11-23T02:32:19.7227740Z 2022-11-23T02:32:19.7227792Z OK 2022-11-23T02:32:19.7227797Z 2022-11-23T02:32:19.7227873Z Generating XML reports... 2022-11-23T02:32:19.7228217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023058.xml 2022-11-23T02:32:19.7228454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7228728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7228847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7229134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7229262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7229270Z 2022-11-23T02:32:19.7229335Z Running tests... 2022-11-23T02:32:19.7229532Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7229758Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22016 2022-11-23T02:32:19.7229909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22017 2022-11-23T02:32:19.7230093Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7230367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7230485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7230768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7230897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7231069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7231344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7231460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7231742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7231870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7232072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7232372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7232671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7232829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7232987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7233047Z ok (3.951s) 2022-11-23T02:32:19.7233052Z 2022-11-23T02:32:19.7233251Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7233317Z Ran 1 test in 3.952s 2022-11-23T02:32:19.7233322Z 2022-11-23T02:32:19.7233374Z OK 2022-11-23T02:32:19.7233381Z 2022-11-23T02:32:19.7233457Z Generating XML reports... 2022-11-23T02:32:19.7233798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023105.xml 2022-11-23T02:32:19.7234030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7234332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7234451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7234736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7234863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7234868Z 2022-11-23T02:32:19.7234934Z Running tests... 2022-11-23T02:32:19.7235128Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7235376Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22199 2022-11-23T02:32:19.7235524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22200 2022-11-23T02:32:19.7235716Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7235991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7236161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7236447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7236575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7236745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7237023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7237139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7237422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7237549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7237720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7238022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7238323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7238483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7238776Z STAGE:2022-11-23 02:31:14 22200:22200 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7238930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7239185Z STAGE:2022-11-23 02:31:15 22199:22199 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7239438Z STAGE:2022-11-23 02:31:15 22199:22199 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7239703Z STAGE:2022-11-23 02:31:15 22199:22199 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7239955Z STAGE:2022-11-23 02:31:15 22200:22200 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7240215Z STAGE:2022-11-23 02:31:15 22200:22200 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7240275Z ok (3.841s) 2022-11-23T02:32:19.7240280Z 2022-11-23T02:32:19.7240478Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7240546Z Ran 1 test in 3.841s 2022-11-23T02:32:19.7240551Z 2022-11-23T02:32:19.7240603Z OK 2022-11-23T02:32:19.7240607Z 2022-11-23T02:32:19.7240684Z Generating XML reports... 2022-11-23T02:32:19.7241061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023112.xml 2022-11-23T02:32:19.7241296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7241572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7241691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7241975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7242103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7242111Z 2022-11-23T02:32:19.7242177Z Running tests... 2022-11-23T02:32:19.7242372Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7242617Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22388 2022-11-23T02:32:19.7242769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22389 2022-11-23T02:32:19.7242957Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7243230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7243343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7243627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7243756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7243927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7244203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7244319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7244601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7244727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7244898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7245198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7245534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7245693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7245949Z STAGE:2022-11-23 02:31:21 22389:22389 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7246107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7246361Z STAGE:2022-11-23 02:31:21 22388:22388 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:32:19.7246616Z STAGE:2022-11-23 02:31:21 22388:22388 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7246878Z STAGE:2022-11-23 02:31:21 22388:22388 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7247129Z STAGE:2022-11-23 02:31:21 22389:22389 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:32:19.7247391Z STAGE:2022-11-23 02:31:21 22389:22389 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:32:19.7247450Z ok (3.939s) 2022-11-23T02:32:19.7247455Z 2022-11-23T02:32:19.7247652Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7247719Z Ran 1 test in 3.940s 2022-11-23T02:32:19.7247756Z 2022-11-23T02:32:19.7247805Z OK 2022-11-23T02:32:19.7247814Z 2022-11-23T02:32:19.7247886Z Generating XML reports... 2022-11-23T02:32:19.7248230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023119.xml 2022-11-23T02:32:19.7248463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7248738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7248857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7249143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7249271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7249277Z 2022-11-23T02:32:19.7249343Z Running tests... 2022-11-23T02:32:19.7249539Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7249770Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22577 2022-11-23T02:32:19.7249919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22578 2022-11-23T02:32:19.7250108Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7250383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7250502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7250782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7250910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7251082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7251359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7251474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7251756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7251885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7252056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7252391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7252688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7252850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7253007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7253068Z ok (3.940s) 2022-11-23T02:32:19.7253073Z 2022-11-23T02:32:19.7253269Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7253337Z Ran 1 test in 3.941s 2022-11-23T02:32:19.7253342Z 2022-11-23T02:32:19.7253394Z OK 2022-11-23T02:32:19.7253398Z 2022-11-23T02:32:19.7253474Z Generating XML reports... 2022-11-23T02:32:19.7253814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023126.xml 2022-11-23T02:32:19.7254048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7254326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7254490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7254778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7254906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7254911Z 2022-11-23T02:32:19.7254976Z Running tests... 2022-11-23T02:32:19.7255171Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7255406Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22922 2022-11-23T02:32:19.7255558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22923 2022-11-23T02:32:19.7255747Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7256022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7256138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7256414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7256545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7256716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7257015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7257293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7257408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7257690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7257817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7257986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7258288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7258447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7258603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7258696Z ok (4.197s) 2022-11-23T02:32:19.7258701Z 2022-11-23T02:32:19.7258900Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7258968Z Ran 1 test in 4.198s 2022-11-23T02:32:19.7258973Z 2022-11-23T02:32:19.7259026Z OK 2022-11-23T02:32:19.7259030Z 2022-11-23T02:32:19.7259109Z Generating XML reports... 2022-11-23T02:32:19.7259450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023133.xml 2022-11-23T02:32:19.7259680Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7259955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7260073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7260355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7260481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7260491Z 2022-11-23T02:32:19.7260551Z Running tests... 2022-11-23T02:32:19.7260747Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7261010Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23269 2022-11-23T02:32:19.7261163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23270 2022-11-23T02:32:19.7261355Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7261631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7261747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7262033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7262163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7262334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7262611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7262726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7263009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7263136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7263306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7263609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7263912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7264072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7264229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7264288Z ok (6.344s) 2022-11-23T02:32:19.7264293Z 2022-11-23T02:32:19.7264490Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7264559Z Ran 1 test in 6.345s 2022-11-23T02:32:19.7264563Z 2022-11-23T02:32:19.7264611Z OK 2022-11-23T02:32:19.7264620Z 2022-11-23T02:32:19.7264691Z Generating XML reports... 2022-11-23T02:32:19.7265035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023140.xml 2022-11-23T02:32:19.7265303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7265580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7265695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7265980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7266107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7266112Z 2022-11-23T02:32:19.7266176Z Running tests... 2022-11-23T02:32:19.7266370Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7266601Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23462 2022-11-23T02:32:19.7266752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23463 2022-11-23T02:32:19.7266943Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7267217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7267370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7267655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7267784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7267955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7268229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7268346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7268633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7268759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7268927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7269228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7269528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7269686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7269866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0xq6yie 2022-11-23T02:32:19.7270055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0xq6yie/_remote_module_non_scriptable.py 2022-11-23T02:32:19.7270216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7270392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_lj3bb51 2022-11-23T02:32:19.7270579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_lj3bb51/_remote_module_non_scriptable.py 2022-11-23T02:32:19.7270640Z ok (3.840s) 2022-11-23T02:32:19.7270645Z 2022-11-23T02:32:19.7270840Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7270908Z Ran 1 test in 3.841s 2022-11-23T02:32:19.7270913Z 2022-11-23T02:32:19.7270965Z OK 2022-11-23T02:32:19.7270970Z 2022-11-23T02:32:19.7271046Z Generating XML reports... 2022-11-23T02:32:19.7271389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023150.xml 2022-11-23T02:32:19.7271621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7271931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7272049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7272335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7272462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7272467Z 2022-11-23T02:32:19.7272531Z Running tests... 2022-11-23T02:32:19.7272726Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7272944Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23649 2022-11-23T02:32:19.7273093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23650 2022-11-23T02:32:19.7273282Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7273559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7273675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7273987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7274116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7274285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7274587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7274862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7274981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7275267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7275396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7275568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7275866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7276024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7276236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7276415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tx7b32b 2022-11-23T02:32:19.7276602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tx7b32b/_remote_module_non_scriptable.py 2022-11-23T02:32:19.7276780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrvcdwaf 2022-11-23T02:32:19.7276966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrvcdwaf/_remote_module_non_scriptable.py 2022-11-23T02:32:19.7277027Z ok (4.155s) 2022-11-23T02:32:19.7277032Z 2022-11-23T02:32:19.7277227Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7277291Z Ran 1 test in 4.156s 2022-11-23T02:32:19.7277300Z 2022-11-23T02:32:19.7277348Z OK 2022-11-23T02:32:19.7277352Z 2022-11-23T02:32:19.7277428Z Generating XML reports... 2022-11-23T02:32:19.7277767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023157.xml 2022-11-23T02:32:19.7278000Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7278318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7278436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7278721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7278851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7278856Z 2022-11-23T02:32:19.7278922Z Running tests... 2022-11-23T02:32:19.7279117Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7279369Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23832 2022-11-23T02:32:19.7279521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23833 2022-11-23T02:32:19.7279710Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:32:19.7279986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7280102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7280418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7280549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7280719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:32:19.7280996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7281114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7281396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7281523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7281695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:32:19.7281998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7282296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:32:19.7282455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:32:19.7282615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:32:19.7282793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkj392wo3 2022-11-23T02:32:19.7282983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkj392wo3/_remote_module_non_scriptable.py 2022-11-23T02:32:19.7283164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9izw5ck 2022-11-23T02:32:19.7283349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9izw5ck/_remote_module_non_scriptable.py 2022-11-23T02:32:19.7283937Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:32:19.7284519Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:32:19.7284613Z ok (6.464s) 2022-11-23T02:32:19.7284618Z 2022-11-23T02:32:19.7284816Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7284885Z Ran 1 test in 6.465s 2022-11-23T02:32:19.7284889Z 2022-11-23T02:32:19.7284942Z OK 2022-11-23T02:32:19.7284947Z 2022-11-23T02:32:19.7285022Z Generating XML reports... 2022-11-23T02:32:19.7285365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023204.xml 2022-11-23T02:32:19.7285597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7285876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7285992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7286330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7286459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7286465Z 2022-11-23T02:32:19.7286530Z Running tests... 2022-11-23T02:32:19.7286729Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7287077Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:32:19.7287082Z 2022-11-23T02:32:19.7287273Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7287342Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7287347Z 2022-11-23T02:32:19.7287410Z OK (skipped=1) 2022-11-23T02:32:19.7287414Z 2022-11-23T02:32:19.7287491Z Generating XML reports... 2022-11-23T02:32:19.7287828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023213.xml 2022-11-23T02:32:19.7288060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:32:19.7288334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:32:19.7288448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:32:19.7288734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:32:19.7288862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:32:19.7288867Z 2022-11-23T02:32:19.7288930Z Running tests... 2022-11-23T02:32:19.7289125Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7289476Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:32:19.7289482Z 2022-11-23T02:32:19.7289678Z ---------------------------------------------------------------------- 2022-11-23T02:32:19.7289748Z Ran 1 test in 0.002s 2022-11-23T02:32:19.7289753Z 2022-11-23T02:32:19.7289816Z OK (skipped=1) 2022-11-23T02:32:19.7289820Z 2022-11-23T02:32:19.7289897Z Generating XML reports... 2022-11-23T02:32:19.7290236Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023216.xml 2022-11-23T02:32:19.7290240Z 2022-11-23T02:32:19.7290558Z ##[endgroup] 2022-11-23T02:32:19.7290913Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_jdx3nt9e) 2022-11-23T02:32:19.7290960Z 2022-11-23T02:32:19.7291151Z Running distributed/test_pg_wrapper ... [2022-11-23 02:32:19.489658] 2022-11-23T02:32:19.7291579Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:32:19.490094] 2022-11-23T02:34:05.6999554Z 2022-11-23T02:34:05.7000556Z Expand the folded group to see the log file of distributed/test_pg_wrapper 2022-11-23T02:34:05.7004507Z ##[group]PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_pg_wrapper_zqqa7ssf) 2022-11-23T02:34:05.7005826Z 2022-11-23T02:34:05.7006666Z 2022-11-23T02:34:05.7011841Z , <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]> 2022-11-23T02:34:05.7019626Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:34:05.7021130Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:34:05.7022411Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:34:05.7023686Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:34:05.7024980Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:34:05.7026727Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:34:05.7028871Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:34:05.7030617Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:34:05.7031884Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:34:05.7035057Z , <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]> 2022-11-23T02:34:05.7039265Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:34:05.7041029Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:34:05.7042927Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:34:05.7044794Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:34:05.7046708Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:34:05.7049571Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7052221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7054027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7056832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7058427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7059022Z 2022-11-23T02:34:05.7059269Z Running tests... 2022-11-23T02:34:05.7060980Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7062474Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24199 2022-11-23T02:34:05.7063947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24200 2022-11-23T02:34:05.7065147Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24201 2022-11-23T02:34:05.7066339Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24202 2022-11-23T02:34:05.7068055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7069247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7070840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7072090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7073244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:34:05.7075074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7076589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7078192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7079293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7079748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:34:05.7080414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7080893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7081499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7081999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7082453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7083115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7083578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7084201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7084691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7085143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7085656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7086176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7086694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:34:05.7087206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:34:05.7087918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7088665Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7089411Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7090253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7090811Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:34:05.7091303Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:34:05.7091921Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-11-23T02:34:05.7092655Z [E ProcessGroupGloo.cpp:137] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-11-23T02:34:05.7093127Z ok (4.527s) 2022-11-23T02:34:05.7093277Z 2022-11-23T02:34:05.7093578Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7093934Z Ran 1 test in 4.528s 2022-11-23T02:34:05.7094097Z 2022-11-23T02:34:05.7094186Z OK 2022-11-23T02:34:05.7094303Z 2022-11-23T02:34:05.7094529Z Generating XML reports... 2022-11-23T02:34:05.7095305Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023222.xml 2022-11-23T02:34:05.7096031Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7096701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7097169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7097785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7098279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7098513Z 2022-11-23T02:34:05.7098615Z Running tests... 2022-11-23T02:34:05.7099058Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7099641Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24530 2022-11-23T02:34:05.7100236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24531 2022-11-23T02:34:05.7100703Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24532 2022-11-23T02:34:05.7101167Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24533 2022-11-23T02:34:05.7101822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7102291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7102910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7103408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7103854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7104520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7104982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7105602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7106087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7106547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:34:05.7107211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7107711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7108329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7108813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7109270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7109939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7110409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7111026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7111499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7111955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:34:05.7112473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7112993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7113550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:34:05.7114074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:34:05.7114802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7115584Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7116432Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7117189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7117580Z ok (4.334s) 2022-11-23T02:34:05.7117724Z 2022-11-23T02:34:05.7118017Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7118354Z Ran 1 test in 4.335s 2022-11-23T02:34:05.7118511Z 2022-11-23T02:34:05.7118598Z OK 2022-11-23T02:34:05.7118724Z 2022-11-23T02:34:05.7118841Z Generating XML reports... 2022-11-23T02:34:05.7119527Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023230.xml 2022-11-23T02:34:05.7120254Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7120901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7121366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7121986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7122467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7122694Z 2022-11-23T02:34:05.7122792Z Running tests... 2022-11-23T02:34:05.7123227Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7123822Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24861 2022-11-23T02:34:05.7124425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24862 2022-11-23T02:34:05.7124876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24863 2022-11-23T02:34:05.7125327Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24864 2022-11-23T02:34:05.7125978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7126537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7127190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7127766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7128383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:34:05.7129050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7129515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7130137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7130641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7131088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7131760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7132227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7132898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7133384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7133835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:34:05.7134497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7134949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7135566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7136069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7136526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7136956Z skip: Need at least 4 CUDA devices (4.626s) 2022-11-23T02:34:05.7137152Z 2022-11-23T02:34:05.7137446Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7137780Z Ran 1 test in 4.626s 2022-11-23T02:34:05.7137937Z 2022-11-23T02:34:05.7138021Z OK (skipped=1) 2022-11-23T02:34:05.7138170Z 2022-11-23T02:34:05.7138285Z Generating XML reports... 2022-11-23T02:34:05.7138965Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023237.xml 2022-11-23T02:34:05.7139693Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7140352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7140824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7141439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7141919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7142151Z 2022-11-23T02:34:05.7142251Z Running tests... 2022-11-23T02:34:05.7142691Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7143317Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25156 2022-11-23T02:34:05.7143938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25157 2022-11-23T02:34:05.7144416Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25158 2022-11-23T02:34:05.7144952Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25159 2022-11-23T02:34:05.7145620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7146080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7146711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7147217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7147687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:34:05.7148370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7148838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7149457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7149933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7150387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7151103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7151570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7152186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7152665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7153118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7153764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7154228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7154842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7155323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7155774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:34:05.7156268Z skip: Need at least 4 CUDA devices (4.237s) 2022-11-23T02:34:05.7156460Z 2022-11-23T02:34:05.7156760Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7157078Z Ran 1 test in 4.237s 2022-11-23T02:34:05.7157235Z 2022-11-23T02:34:05.7157334Z OK (skipped=1) 2022-11-23T02:34:05.7157482Z 2022-11-23T02:34:05.7157598Z Generating XML reports... 2022-11-23T02:34:05.7158276Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023245.xml 2022-11-23T02:34:05.7159005Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7159668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7160136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7160758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7161232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7161462Z 2022-11-23T02:34:05.7161562Z Running tests... 2022-11-23T02:34:05.7161993Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7162594Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25451 2022-11-23T02:34:05.7163262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25452 2022-11-23T02:34:05.7163723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25453 2022-11-23T02:34:05.7164182Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25454 2022-11-23T02:34:05.7164825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7165290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7165904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7166389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7166842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7167504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7167972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7168572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7169116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7169571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:34:05.7170231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7170691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7171309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7171790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7172233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7172898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7173359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7173973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7174452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7174905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:34:05.7175416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7175939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7176433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:34:05.7176940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:34:05.7177643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7178387Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7179124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7179860Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7180404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:34:05.7180910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:34:05.7181461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T02:34:05.7182163Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:34:05.7182717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T02:34:05.7183413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:34:05.7184151Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:34:05.7184890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:34:05.7185286Z ok (4.528s) 2022-11-23T02:34:05.7185430Z 2022-11-23T02:34:05.7185720Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7186045Z Ran 1 test in 4.529s 2022-11-23T02:34:05.7186203Z 2022-11-23T02:34:05.7186286Z OK 2022-11-23T02:34:05.7186414Z 2022-11-23T02:34:05.7186530Z Generating XML reports... 2022-11-23T02:34:05.7187345Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023252.xml 2022-11-23T02:34:05.7188086Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7188751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7189219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7189827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7190310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7190546Z 2022-11-23T02:34:05.7190646Z Running tests... 2022-11-23T02:34:05.7191081Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7191668Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25794 2022-11-23T02:34:05.7192256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25795 2022-11-23T02:34:05.7192721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25796 2022-11-23T02:34:05.7193165Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25797 2022-11-23T02:34:05.7193811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7194280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7194894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7195383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7195839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7196597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7197095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7197711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7198196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7198646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:34:05.7199306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7199832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7200452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7200938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7201382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:34:05.7202038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7202498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7203109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7203591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7204042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7204567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7205074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7205644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:34:05.7206161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:34:05.7206868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7207615Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7208360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7209098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7209497Z ok (4.536s) 2022-11-23T02:34:05.7209628Z 2022-11-23T02:34:05.7209923Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7210260Z Ran 1 test in 4.536s 2022-11-23T02:34:05.7210423Z 2022-11-23T02:34:05.7210510Z OK 2022-11-23T02:34:05.7210636Z 2022-11-23T02:34:05.7210753Z Generating XML reports... 2022-11-23T02:34:05.7211441Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023259.xml 2022-11-23T02:34:05.7212168Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7212828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7213279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7213895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7214380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7214611Z 2022-11-23T02:34:05.7214711Z Running tests... 2022-11-23T02:34:05.7215153Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7215743Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26125 2022-11-23T02:34:05.7216339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26126 2022-11-23T02:34:05.7216796Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26127 2022-11-23T02:34:05.7217254Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26128 2022-11-23T02:34:05.7217901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7218417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7219039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7219532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7219988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:34:05.7220640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7221102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7221723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7222205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7222667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:34:05.7223333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7223800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7224466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7224933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7225387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7226048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7226505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7227123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7227607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7228060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7228453Z skip: Need at least 4 CUDA devices (4.240s) 2022-11-23T02:34:05.7228645Z 2022-11-23T02:34:05.7228938Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7229272Z Ran 1 test in 4.240s 2022-11-23T02:34:05.7229427Z 2022-11-23T02:34:05.7229525Z OK (skipped=1) 2022-11-23T02:34:05.7229673Z 2022-11-23T02:34:05.7229790Z Generating XML reports... 2022-11-23T02:34:05.7230465Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023307.xml 2022-11-23T02:34:05.7231188Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7231845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7232313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7232930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7233418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7233647Z 2022-11-23T02:34:05.7233747Z Running tests... 2022-11-23T02:34:05.7234182Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7234789Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26420 2022-11-23T02:34:05.7235385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26421 2022-11-23T02:34:05.7235851Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26422 2022-11-23T02:34:05.7236437Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26423 2022-11-23T02:34:05.7237101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7237564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7238187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7238678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7239120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7239783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7240244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7240862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7241347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7241800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:34:05.7242518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7242981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7243585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7244069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7244527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7245186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7245649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7246261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7246743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7247187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:34:05.7247594Z skip: Need at least 4 CUDA devices (4.625s) 2022-11-23T02:34:05.7247783Z 2022-11-23T02:34:05.7248075Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7248412Z Ran 1 test in 4.625s 2022-11-23T02:34:05.7248569Z 2022-11-23T02:34:05.7248668Z OK (skipped=1) 2022-11-23T02:34:05.7248816Z 2022-11-23T02:34:05.7248930Z Generating XML reports... 2022-11-23T02:34:05.7249614Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023314.xml 2022-11-23T02:34:05.7250326Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7250993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7251459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7252077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7252559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7252789Z 2022-11-23T02:34:05.7252887Z Running tests... 2022-11-23T02:34:05.7253320Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7253905Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26715 2022-11-23T02:34:05.7254561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26716 2022-11-23T02:34:05.7255028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26717 2022-11-23T02:34:05.7255487Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26718 2022-11-23T02:34:05.7256144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7256610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7257229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7257711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7258154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7258814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7259277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7259897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7260427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7260887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:34:05.7261550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7262010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7262627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7263106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7263561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7264221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7264683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7265303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7265772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7266222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:34:05.7266735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7267253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7267764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:34:05.7268478Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7269034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:34:05.7269739Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7270467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7271204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:34:05.7271759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:34:05.7272266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:34:05.7272822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T02:34:05.7273333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T02:34:05.7274037Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:34:05.7274772Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:34:05.7275491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:34:05.7276308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:34:05.7276711Z ok (4.654s) 2022-11-23T02:34:05.7276855Z 2022-11-23T02:34:05.7277152Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7277494Z Ran 1 test in 4.654s 2022-11-23T02:34:05.7277654Z 2022-11-23T02:34:05.7277736Z OK 2022-11-23T02:34:05.7277862Z 2022-11-23T02:34:05.7277979Z Generating XML reports... 2022-11-23T02:34:05.7278700Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023322.xml 2022-11-23T02:34:05.7279439Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7280109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7280579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7281203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7281692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7281927Z 2022-11-23T02:34:05.7282027Z Running tests... 2022-11-23T02:34:05.7282451Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7283033Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27058 2022-11-23T02:34:05.7283613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27059 2022-11-23T02:34:05.7284272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7284742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7285359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7285843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7286297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7286800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7287471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7287947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7288569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7289058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7289517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7290030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7290730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7291535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7292074Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:34:05.7292562Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:34:05.7292913Z ok (3.515s) 2022-11-23T02:34:05.7293056Z 2022-11-23T02:34:05.7293350Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7293690Z Ran 1 test in 3.515s 2022-11-23T02:34:05.7293849Z 2022-11-23T02:34:05.7293919Z OK 2022-11-23T02:34:05.7294045Z 2022-11-23T02:34:05.7294160Z Generating XML reports... 2022-11-23T02:34:05.7294841Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023329.xml 2022-11-23T02:34:05.7295562Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7296229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7296695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7297494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7297987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7298205Z 2022-11-23T02:34:05.7298305Z Running tests... 2022-11-23T02:34:05.7298743Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7299335Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27245 2022-11-23T02:34:05.7299929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27246 2022-11-23T02:34:05.7300586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7301056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7301672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7302146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7302605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7303118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7303803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7304266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7304892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7305380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7305827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7306340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7307047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7307792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7308199Z ok (4.520s) 2022-11-23T02:34:05.7308341Z 2022-11-23T02:34:05.7308631Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7308967Z Ran 1 test in 4.520s 2022-11-23T02:34:05.7309127Z 2022-11-23T02:34:05.7309209Z OK 2022-11-23T02:34:05.7309373Z 2022-11-23T02:34:05.7309493Z Generating XML reports... 2022-11-23T02:34:05.7310174Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023336.xml 2022-11-23T02:34:05.7310900Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7311572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7312035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7312654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7313142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7313372Z 2022-11-23T02:34:05.7313458Z Running tests... 2022-11-23T02:34:05.7313891Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7314503Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27446 2022-11-23T02:34:05.7315111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27447 2022-11-23T02:34:05.7315803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7316893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7317521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7317998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7318452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7319118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7319582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7320198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7320681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7321137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7321650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7322155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7322865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7323607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7324167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:34:05.7324682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:34:05.7325388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:34:05.7326123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:34:05.7326519Z ok (4.518s) 2022-11-23T02:34:05.7326649Z 2022-11-23T02:34:05.7326940Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7327277Z Ran 1 test in 4.518s 2022-11-23T02:34:05.7327436Z 2022-11-23T02:34:05.7327518Z OK 2022-11-23T02:34:05.7327645Z 2022-11-23T02:34:05.7327763Z Generating XML reports... 2022-11-23T02:34:05.7328448Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023344.xml 2022-11-23T02:34:05.7329272Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7329922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7330394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7331014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7331503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7331733Z 2022-11-23T02:34:05.7331831Z Running tests... 2022-11-23T02:34:05.7332267Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7332848Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27657 2022-11-23T02:34:05.7333442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27658 2022-11-23T02:34:05.7334085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7334603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7335225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7335712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7336168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7336685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7337361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7337812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7338429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7338919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7339373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7339888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7340604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7341346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7341746Z ok (4.119s) 2022-11-23T02:34:05.7341876Z 2022-11-23T02:34:05.7342168Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7342505Z Ran 1 test in 4.120s 2022-11-23T02:34:05.7342663Z 2022-11-23T02:34:05.7342746Z OK 2022-11-23T02:34:05.7342872Z 2022-11-23T02:34:05.7342988Z Generating XML reports... 2022-11-23T02:34:05.7343671Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023351.xml 2022-11-23T02:34:05.7344395Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:34:05.7345041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7345509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7346122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7346610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7346892Z 2022-11-23T02:34:05.7346991Z Running tests... 2022-11-23T02:34:05.7347432Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7348029Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27858 2022-11-23T02:34:05.7348635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27859 2022-11-23T02:34:05.7349277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7349745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7350365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7350856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7351314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:34:05.7351983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:05.7352450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:05.7353093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:05.7353583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:05.7354038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:34:05.7354557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:34:05.7355071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:34:05.7355784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7356597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:34:05.7357155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:34:05.7357652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:34:05.7358365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:34:05.7359110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:34:05.7359517Z ok (4.316s) 2022-11-23T02:34:05.7359661Z 2022-11-23T02:34:05.7359953Z ---------------------------------------------------------------------- 2022-11-23T02:34:05.7360291Z Ran 1 test in 4.317s 2022-11-23T02:34:05.7360449Z 2022-11-23T02:34:05.7360535Z OK 2022-11-23T02:34:05.7360665Z 2022-11-23T02:34:05.7360770Z Generating XML reports... 2022-11-23T02:34:05.7361456Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023358.xml 2022-11-23T02:34:05.7361854Z 2022-11-23T02:34:05.7362320Z ##[endgroup] 2022-11-23T02:34:05.7362943Z FINISHED PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_pg_wrapper_zqqa7ssf) 2022-11-23T02:34:05.7363289Z 2022-11-23T02:34:05.7363581Z Running distributed/test_multi_threaded_pg ... [2022-11-23 02:34:05.702934] 2022-11-23T02:34:05.7364395Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:34:05.703811] 2022-11-23T02:34:09.1249991Z 2022-11-23T02:34:09.1251197Z Expand the folded group to see the log file of distributed/test_multi_threaded_pg 2022-11-23T02:34:09.1255731Z ##[group]PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_multi_threaded_pg_r9jgzbch) 2022-11-23T02:34:09.1257850Z Test results will be stored in test-reports/python-unittest/distributed.test_multi_threaded_pg 2022-11-23T02:34:09.1258587Z 2022-11-23T02:34:09.1258835Z Running tests... 2022-11-23T02:34:09.1259985Z ---------------------------------------------------------------------- 2022-11-23T02:34:09.1262056Z test_allgather (__main__.TestCollectivesWithBaseClass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.482s) 2022-11-23T02:34:09.1264497Z test_broadcast (__main__.TestCollectivesWithBaseClass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:34:09.1266985Z test_broadcast_object_list (__main__.TestCollectivesWithBaseClass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:34:09.1269486Z test_reduce_scatter (__main__.TestCollectivesWithBaseClass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:34:09.1272147Z test_scatter (__main__.TestCollectivesWithBaseClass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:34:09.1274634Z test_broadcast_object_list (__main__.TestCollectivesWithWrapper) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:34:09.1275654Z 2022-11-23T02:34:09.1276536Z ---------------------------------------------------------------------- 2022-11-23T02:34:09.1277376Z Ran 6 tests in 0.485s 2022-11-23T02:34:09.1277772Z 2022-11-23T02:34:09.1278010Z OK (skipped=6) 2022-11-23T02:34:09.1278387Z 2022-11-23T02:34:09.1278671Z Generating XML reports... 2022-11-23T02:34:09.1280462Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20221123023407.xml 2022-11-23T02:34:09.1282831Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20221123023407.xml 2022-11-23T02:34:09.1283826Z 2022-11-23T02:34:09.1284594Z ##[endgroup] 2022-11-23T02:34:09.1286238Z FINISHED PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_multi_threaded_pg_r9jgzbch) 2022-11-23T02:34:09.1287155Z 2022-11-23T02:34:09.1287934Z Running distributed/test_dynamo_distributed ... [2022-11-23 02:34:09.125460] 2022-11-23T02:34:09.1290046Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:34:09.126314] 2022-11-23T02:34:12.5241675Z 2022-11-23T02:34:12.5243371Z Expand the folded group to see the log file of distributed/test_dynamo_distributed 2022-11-23T02:34:12.5245561Z ##[group]PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_dynamo_distributed_304pkg3f) 2022-11-23T02:34:12.5245850Z 2022-11-23T02:34:12.5246095Z ##[endgroup] 2022-11-23T02:34:12.5246790Z FINISHED PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_dynamo_distributed_304pkg3f) 2022-11-23T02:34:12.5247077Z 2022-11-23T02:34:12.5251331Z Running distributed/test_c10d_spawn_ucc ... [2022-11-23 02:34:12.524730] 2022-11-23T02:34:12.5260454Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:34:12.525592] 2022-11-23T02:34:46.6888723Z 2022-11-23T02:34:46.6890332Z Expand the folded group to see the log file of distributed/test_c10d_spawn_ucc 2022-11-23T02:34:46.6893239Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_ucc_cl4rg31e) 2022-11-23T02:34:46.6894828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmz3s62z 2022-11-23T02:34:46.6896295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmz3s62z/_remote_module_non_scriptable.py 2022-11-23T02:34:46.6898511Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_ucc>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_ucc>]> 2022-11-23T02:34:46.6900467Z test_shared_allgather_ucc (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:34:46.6901857Z test_shared_allreduce_ucc (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:34:46.6903109Z test_shared_broadcast_ucc (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:34:46.6904116Z 2022-11-23T02:34:46.6904944Z 2022-11-23T02:34:46.6907812Z , <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_reduce>]> 2022-11-23T02:34:46.6910430Z test_all_gather (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:34:46.6911469Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:34:46.6912497Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:34:46.6913755Z test_allreduce (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:34:46.6914772Z test_broadcast (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:34:46.6915772Z test_reduce (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:34:46.6917846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:46.6919039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:46.6920647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:46.6921902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:46.6923143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzw540eq 2022-11-23T02:34:46.6924585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzw540eq/_remote_module_non_scriptable.py 2022-11-23T02:34:46.6926244Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:34:46.6926940Z 2022-11-23T02:34:46.6927184Z Running tests... 2022-11-23T02:34:46.6928313Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.6929555Z test_shared_allgather_ucc (__main__.ProcessGroupShareTensorTest) ... skip: UCC needed (0.000s) 2022-11-23T02:34:46.6930287Z 2022-11-23T02:34:46.6931017Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.6931871Z Ran 1 test in 0.001s 2022-11-23T02:34:46.6932262Z 2022-11-23T02:34:46.6932498Z OK (skipped=1) 2022-11-23T02:34:46.6932870Z 2022-11-23T02:34:46.6933153Z Generating XML reports... 2022-11-23T02:34:46.6934893Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123023416.xml 2022-11-23T02:34:46.6936916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:46.6938113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:46.6939685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:46.6941107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:46.6942336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hj73e3a 2022-11-23T02:34:46.6943768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hj73e3a/_remote_module_non_scriptable.py 2022-11-23T02:34:46.6945404Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:34:46.6946098Z 2022-11-23T02:34:46.6946338Z Running tests... 2022-11-23T02:34:46.6947438Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.6948672Z test_shared_allreduce_ucc (__main__.ProcessGroupShareTensorTest) ... skip: UCC needed (0.000s) 2022-11-23T02:34:46.6949393Z 2022-11-23T02:34:46.6950115Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.6950956Z Ran 1 test in 0.001s 2022-11-23T02:34:46.6951347Z 2022-11-23T02:34:46.6951580Z OK (skipped=1) 2022-11-23T02:34:46.6951951Z 2022-11-23T02:34:46.6952218Z Generating XML reports... 2022-11-23T02:34:46.6953951Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123023419.xml 2022-11-23T02:34:46.6956223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:46.6957421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:46.6958998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:46.6960240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:46.6961474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsipawsm6 2022-11-23T02:34:46.6962901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsipawsm6/_remote_module_non_scriptable.py 2022-11-23T02:34:46.6964565Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:34:46.6965264Z 2022-11-23T02:34:46.6965506Z Running tests... 2022-11-23T02:34:46.6966596Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.6967823Z test_shared_broadcast_ucc (__main__.ProcessGroupShareTensorTest) ... skip: UCC needed (0.000s) 2022-11-23T02:34:46.6968544Z 2022-11-23T02:34:46.6969267Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.6970103Z Ran 1 test in 0.001s 2022-11-23T02:34:46.6970488Z 2022-11-23T02:34:46.6970707Z OK (skipped=1) 2022-11-23T02:34:46.6971075Z 2022-11-23T02:34:46.6971356Z Generating XML reports... 2022-11-23T02:34:46.6973096Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123023423.xml 2022-11-23T02:34:46.6975127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:46.6976313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:46.6977879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:46.6979128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:46.6980347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tl29bmp 2022-11-23T02:34:46.6981766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tl29bmp/_remote_module_non_scriptable.py 2022-11-23T02:34:46.6983410Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:34:46.6984112Z 2022-11-23T02:34:46.6984353Z Running tests... 2022-11-23T02:34:46.6985443Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.6986886Z test_all_gather (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:34:46.6987672Z 2022-11-23T02:34:46.6988400Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.6989237Z Ran 1 test in 0.001s 2022-11-23T02:34:46.6989626Z 2022-11-23T02:34:46.6989855Z OK (skipped=1) 2022-11-23T02:34:46.6990224Z 2022-11-23T02:34:46.6990508Z Generating XML reports... 2022-11-23T02:34:46.6992260Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123023426.xml 2022-11-23T02:34:46.6994281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:46.6995459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:46.6997170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:46.6998419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:46.6999268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccu5o374 2022-11-23T02:34:46.6999705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccu5o374/_remote_module_non_scriptable.py 2022-11-23T02:34:46.7000258Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:34:46.7000471Z 2022-11-23T02:34:46.7000551Z Running tests... 2022-11-23T02:34:46.7000890Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7001288Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:34:46.7001529Z 2022-11-23T02:34:46.7001747Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7002007Z Ran 1 test in 0.001s 2022-11-23T02:34:46.7002116Z 2022-11-23T02:34:46.7002196Z OK (skipped=1) 2022-11-23T02:34:46.7002311Z 2022-11-23T02:34:46.7002401Z Generating XML reports... 2022-11-23T02:34:46.7002933Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123023430.xml 2022-11-23T02:34:46.7003545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:46.7003906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:46.7004385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:46.7004761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:46.7005122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtccmg4m 2022-11-23T02:34:46.7005558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtccmg4m/_remote_module_non_scriptable.py 2022-11-23T02:34:46.7006061Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:34:46.7006272Z 2022-11-23T02:34:46.7006350Z Running tests... 2022-11-23T02:34:46.7006690Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7007099Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:34:46.7007460Z 2022-11-23T02:34:46.7007665Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7007889Z Ran 1 test in 0.001s 2022-11-23T02:34:46.7008002Z 2022-11-23T02:34:46.7008074Z OK (skipped=1) 2022-11-23T02:34:46.7008185Z 2022-11-23T02:34:46.7008270Z Generating XML reports... 2022-11-23T02:34:46.7008757Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123023433.xml 2022-11-23T02:34:46.7009321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:46.7009707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:46.7010152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:46.7010491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:46.7010842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8vdy_t0 2022-11-23T02:34:46.7011256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8vdy_t0/_remote_module_non_scriptable.py 2022-11-23T02:34:46.7011733Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:34:46.7011931Z 2022-11-23T02:34:46.7012004Z Running tests... 2022-11-23T02:34:46.7012318Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7012687Z test_allreduce (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:34:46.7012912Z 2022-11-23T02:34:46.7013114Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7013342Z Ran 1 test in 0.001s 2022-11-23T02:34:46.7013455Z 2022-11-23T02:34:46.7013529Z OK (skipped=1) 2022-11-23T02:34:46.7013636Z 2022-11-23T02:34:46.7013757Z Generating XML reports... 2022-11-23T02:34:46.7014250Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123023436.xml 2022-11-23T02:34:46.7014816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:46.7015152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:46.7015590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:46.7015931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:46.7016293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps86rx27c 2022-11-23T02:34:46.7016708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps86rx27c/_remote_module_non_scriptable.py 2022-11-23T02:34:46.7017179Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:34:46.7017384Z 2022-11-23T02:34:46.7017464Z Running tests... 2022-11-23T02:34:46.7017791Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7018174Z test_broadcast (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:34:46.7018405Z 2022-11-23T02:34:46.7018592Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7018846Z Ran 1 test in 0.001s 2022-11-23T02:34:46.7018966Z 2022-11-23T02:34:46.7019047Z OK (skipped=1) 2022-11-23T02:34:46.7019164Z 2022-11-23T02:34:46.7019251Z Generating XML reports... 2022-11-23T02:34:46.7019741Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123023440.xml 2022-11-23T02:34:46.7020308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:34:46.7020642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:34:46.7021074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:34:46.7021422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:34:46.7021770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25epsqeg 2022-11-23T02:34:46.7022173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25epsqeg/_remote_module_non_scriptable.py 2022-11-23T02:34:46.7022634Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:34:46.7022868Z 2022-11-23T02:34:46.7022942Z Running tests... 2022-11-23T02:34:46.7023256Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7023612Z test_reduce (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:34:46.7023833Z 2022-11-23T02:34:46.7024031Z ---------------------------------------------------------------------- 2022-11-23T02:34:46.7024272Z Ran 1 test in 0.001s 2022-11-23T02:34:46.7024385Z 2022-11-23T02:34:46.7024455Z OK (skipped=1) 2022-11-23T02:34:46.7024562Z 2022-11-23T02:34:46.7024645Z Generating XML reports... 2022-11-23T02:34:46.7025133Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123023443.xml 2022-11-23T02:34:46.7025412Z 2022-11-23T02:34:46.7025675Z ##[endgroup] 2022-11-23T02:34:46.7026121Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_ucc_cl4rg31e) 2022-11-23T02:34:46.7026370Z 2022-11-23T02:34:46.7026572Z Running distributed/test_c10d_spawn_gloo ... [2022-11-23 02:34:46.690225] 2022-11-23T02:34:46.7027214Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:34:46.691100] 2022-11-23T02:36:37.7885965Z 2022-11-23T02:36:37.7887153Z Expand the folded group to see the log file of distributed/test_c10d_spawn_gloo 2022-11-23T02:36:37.7890715Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_gloo_ddk8dp0n) 2022-11-23T02:36:37.7894799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkpspzwy 2022-11-23T02:36:37.7896211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkpspzwy/_remote_module_non_scriptable.py 2022-11-23T02:36:37.7897423Z , <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]> 2022-11-23T02:36:37.7898579Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) 2022-11-23T02:36:37.7899201Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) 2022-11-23T02:36:37.7899810Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) 2022-11-23T02:36:37.7900992Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]> 2022-11-23T02:36:37.7902095Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:36:37.7902797Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:36:37.7903671Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:36:37.7904666Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:36:37.7905388Z 2022-11-23T02:36:37.7906111Z 2022-11-23T02:36:37.7908503Z , <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]> 2022-11-23T02:36:37.7911302Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:36:37.7912110Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:36:37.7912949Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:36:37.7913825Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:36:37.7914634Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:36:37.7915354Z test_gather (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:36:37.7916242Z test_reduce (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:36:37.7917537Z test_scatter (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:36:37.7919626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.7920462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.7921574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.7922447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.7923273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjb1p4qci 2022-11-23T02:36:37.7924435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjb1p4qci/_remote_module_non_scriptable.py 2022-11-23T02:36:37.7925590Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.7926086Z 2022-11-23T02:36:37.7926271Z Running tests... 2022-11-23T02:36:37.7927046Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.7928141Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:37.7929017Z ok (0.025s) 2022-11-23T02:36:37.7929240Z 2022-11-23T02:36:37.7929754Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.7930347Z Ran 1 test in 0.025s 2022-11-23T02:36:37.7930622Z 2022-11-23T02:36:37.7930769Z OK 2022-11-23T02:36:37.7930989Z 2022-11-23T02:36:37.7931203Z Generating XML reports... 2022-11-23T02:36:37.7932535Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123023450.xml 2022-11-23T02:36:37.7934018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.7934835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.7935902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.7936760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.7937617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs2ftti4 2022-11-23T02:36:37.7938614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs2ftti4/_remote_module_non_scriptable.py 2022-11-23T02:36:37.7939758Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.7940247Z 2022-11-23T02:36:37.7940434Z Running tests... 2022-11-23T02:36:37.7941202Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.7942281Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:37.7943146Z ok (1.986s) 2022-11-23T02:36:37.7943399Z 2022-11-23T02:36:37.7943895Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.7944476Z Ran 1 test in 1.986s 2022-11-23T02:36:37.7944748Z 2022-11-23T02:36:37.7944894Z OK 2022-11-23T02:36:37.7945116Z 2022-11-23T02:36:37.7945320Z Generating XML reports... 2022-11-23T02:36:37.7946745Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123023454.xml 2022-11-23T02:36:37.7948481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.7949307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.7950392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.7951248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.7952100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8yye44h 2022-11-23T02:36:37.7953101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8yye44h/_remote_module_non_scriptable.py 2022-11-23T02:36:37.7954234Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.7954726Z 2022-11-23T02:36:37.7954900Z Running tests... 2022-11-23T02:36:37.7955635Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.7956975Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:37.7957840Z ok (3.629s) 2022-11-23T02:36:37.7958095Z 2022-11-23T02:36:37.7958606Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.7959183Z Ran 1 test in 3.630s 2022-11-23T02:36:37.7959460Z 2022-11-23T02:36:37.7959610Z OK 2022-11-23T02:36:37.7959835Z 2022-11-23T02:36:37.7960043Z Generating XML reports... 2022-11-23T02:36:37.7961330Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123023500.xml 2022-11-23T02:36:37.7962828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.7963653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.7964737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.7965592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.7966450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0k_p_5f3 2022-11-23T02:36:37.7967435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0k_p_5f3/_remote_module_non_scriptable.py 2022-11-23T02:36:37.7968536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.7969014Z 2022-11-23T02:36:37.7969188Z Running tests... 2022-11-23T02:36:37.7969953Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.7971409Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.7972427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.7973516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.7974361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.7975207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hu3e5c3 2022-11-23T02:36:37.7976160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hu3e5c3/_remote_module_non_scriptable.py 2022-11-23T02:36:37.7977387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.7978200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.7979273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.7980219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.7981067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6nj1_z4 2022-11-23T02:36:37.7982052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6nj1_z4/_remote_module_non_scriptable.py 2022-11-23T02:36:37.7982710Z ok (3.496s) 2022-11-23T02:36:37.7982962Z 2022-11-23T02:36:37.7983468Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.7984052Z Ran 1 test in 3.497s 2022-11-23T02:36:37.7984330Z 2022-11-23T02:36:37.7984481Z OK 2022-11-23T02:36:37.7984704Z 2022-11-23T02:36:37.7984914Z Generating XML reports... 2022-11-23T02:36:37.7986128Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123023507.xml 2022-11-23T02:36:37.7987502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.7988307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.7989384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.7990309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.7991155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8i1pp2k 2022-11-23T02:36:37.7992155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8i1pp2k/_remote_module_non_scriptable.py 2022-11-23T02:36:37.7993294Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.7993782Z 2022-11-23T02:36:37.7993959Z Running tests... 2022-11-23T02:36:37.7994691Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.7996240Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.7997244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.7998339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.7999204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8000062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0f80s65 2022-11-23T02:36:37.8001051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0f80s65/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8002265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8003052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8004145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8005008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8005874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdcv5ip8 2022-11-23T02:36:37.8006889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdcv5ip8/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8007594Z ok (4.153s) 2022-11-23T02:36:37.8007858Z 2022-11-23T02:36:37.8008371Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8008947Z Ran 1 test in 4.154s 2022-11-23T02:36:37.8009226Z 2022-11-23T02:36:37.8009381Z OK 2022-11-23T02:36:37.8009609Z 2022-11-23T02:36:37.8009818Z Generating XML reports... 2022-11-23T02:36:37.8011031Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123023514.xml 2022-11-23T02:36:37.8012369Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:36:37.8013650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8014473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8015534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8016397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8017264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2p0wpj2 2022-11-23T02:36:37.8018269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2p0wpj2/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8019408Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8019902Z 2022-11-23T02:36:37.8020083Z Running tests... 2022-11-23T02:36:37.8020849Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8022372Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8023388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8024476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8025325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8026182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb423719j 2022-11-23T02:36:37.8027167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb423719j/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8028387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8029204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8030257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8031128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8031974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_bj_kvy 2022-11-23T02:36:37.8032951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_bj_kvy/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8033634Z ok (4.019s) 2022-11-23T02:36:37.8033888Z 2022-11-23T02:36:37.8034393Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8034984Z Ran 1 test in 4.020s 2022-11-23T02:36:37.8035234Z 2022-11-23T02:36:37.8035389Z OK 2022-11-23T02:36:37.8035625Z 2022-11-23T02:36:37.8035834Z Generating XML reports... 2022-11-23T02:36:37.8037164Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123023522.xml 2022-11-23T02:36:37.8038402Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:36:37.8039008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8039410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8039925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8040315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8040728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz00zvjuk 2022-11-23T02:36:37.8041254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz00zvjuk/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8041787Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8042014Z 2022-11-23T02:36:37.8042104Z Running tests... 2022-11-23T02:36:37.8042471Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8043142Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8043626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8044115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8044528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8044932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8eb5_9v 2022-11-23T02:36:37.8045412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8eb5_9v/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8045991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8046429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8047048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8047519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8047899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtnwl7ux 2022-11-23T02:36:37.8048348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtnwl7ux/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8048671Z ok (4.005s) 2022-11-23T02:36:37.8048788Z 2022-11-23T02:36:37.8049018Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8049292Z Ran 1 test in 4.006s 2022-11-23T02:36:37.8049423Z 2022-11-23T02:36:37.8049496Z OK 2022-11-23T02:36:37.8049580Z 2022-11-23T02:36:37.8049679Z Generating XML reports... 2022-11-23T02:36:37.8050225Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123023529.xml 2022-11-23T02:36:37.8050763Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:36:37.8051332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8051704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8052187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8052567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8052933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5rbe2rv 2022-11-23T02:36:37.8053376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5rbe2rv/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8053883Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8054105Z 2022-11-23T02:36:37.8054192Z Running tests... 2022-11-23T02:36:37.8054541Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8054989Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29763 2022-11-23T02:36:37.8055438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29764 2022-11-23T02:36:37.8055946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8056338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8056829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8057216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8057599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwa6d_356 2022-11-23T02:36:37.8058046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwa6d_356/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8058592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8058961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8059423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8059816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8060207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknwqzzd8 2022-11-23T02:36:37.8060649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknwqzzd8/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8061101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:37.8061498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:37.8061913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:37.8062304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:37.8062858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8063436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8063768Z ok (3.965s) 2022-11-23T02:36:37.8063888Z 2022-11-23T02:36:37.8064118Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8064393Z Ran 1 test in 3.966s 2022-11-23T02:36:37.8064521Z 2022-11-23T02:36:37.8064600Z OK 2022-11-23T02:36:37.8064710Z 2022-11-23T02:36:37.8064788Z Generating XML reports... 2022-11-23T02:36:37.8065331Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123023537.xml 2022-11-23T02:36:37.8065961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8066333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8066815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8067203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8067589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnn2kcg3 2022-11-23T02:36:37.8068035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnn2kcg3/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8068525Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8068739Z 2022-11-23T02:36:37.8068827Z Running tests... 2022-11-23T02:36:37.8069177Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8069623Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29954 2022-11-23T02:36:37.8070065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29955 2022-11-23T02:36:37.8070580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8071023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8071490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8071884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8072274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenseue50 2022-11-23T02:36:37.8072726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenseue50/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8073270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8073641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8074125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8074492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8074875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprylz60tz 2022-11-23T02:36:37.8075322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprylz60tz/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8075789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:37.8076270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:37.8076874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:37.8077478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8077957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:37.8078530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8078881Z ok (3.942s) 2022-11-23T02:36:37.8079009Z 2022-11-23T02:36:37.8079258Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8079555Z Ran 1 test in 3.943s 2022-11-23T02:36:37.8079701Z 2022-11-23T02:36:37.8079782Z OK 2022-11-23T02:36:37.8079898Z 2022-11-23T02:36:37.8080006Z Generating XML reports... 2022-11-23T02:36:37.8080595Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123023544.xml 2022-11-23T02:36:37.8081235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8081639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8082160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8082579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8082996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzexrxhwz 2022-11-23T02:36:37.8083475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzexrxhwz/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8084021Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8084255Z 2022-11-23T02:36:37.8084324Z Running tests... 2022-11-23T02:36:37.8084698Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8085189Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30145 2022-11-23T02:36:37.8085677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30146 2022-11-23T02:36:37.8086226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8086671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8087316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8087835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8088205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyssljmx 2022-11-23T02:36:37.8088653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyssljmx/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8089204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8089580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8090070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8090471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8090865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtdw_cvn 2022-11-23T02:36:37.8091331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtdw_cvn/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8091758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:37.8092158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:37.8092567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:37.8092987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:37.8093557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8094152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8094483Z ok (4.054s) 2022-11-23T02:36:37.8094591Z 2022-11-23T02:36:37.8094820Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8095105Z Ran 1 test in 4.055s 2022-11-23T02:36:37.8095248Z 2022-11-23T02:36:37.8095322Z OK 2022-11-23T02:36:37.8095433Z 2022-11-23T02:36:37.8095536Z Generating XML reports... 2022-11-23T02:36:37.8096096Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123023552.xml 2022-11-23T02:36:37.8096724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8097078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8097571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8097974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8098362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprsfhb1ff 2022-11-23T02:36:37.8098817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprsfhb1ff/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8099328Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8099545Z 2022-11-23T02:36:37.8099635Z Running tests... 2022-11-23T02:36:37.8099981Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8100412Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30336 2022-11-23T02:36:37.8101036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30337 2022-11-23T02:36:37.8101615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8101993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8102494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8102896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8103295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2xm39se 2022-11-23T02:36:37.8103729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2xm39se/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8104282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8104666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8105166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8105570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8105971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s777xw6 2022-11-23T02:36:37.8106462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s777xw6/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8106869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:37.8107268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:37.8107684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:37.8122198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8122820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:37.8123506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8123862Z ok (4.165s) 2022-11-23T02:36:37.8123992Z 2022-11-23T02:36:37.8124251Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8124553Z Ran 1 test in 4.166s 2022-11-23T02:36:37.8124690Z 2022-11-23T02:36:37.8124769Z OK 2022-11-23T02:36:37.8124881Z 2022-11-23T02:36:37.8124984Z Generating XML reports... 2022-11-23T02:36:37.8125571Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123023559.xml 2022-11-23T02:36:37.8126219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8126612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8127253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8127663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8128064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qcqak_z 2022-11-23T02:36:37.8128532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qcqak_z/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8129055Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8129283Z 2022-11-23T02:36:37.8129374Z Running tests... 2022-11-23T02:36:37.8129711Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8130170Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30527 2022-11-23T02:36:37.8130796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30528 2022-11-23T02:36:37.8131475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8131863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8132373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8132781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8133158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpti6h0c7i 2022-11-23T02:36:37.8133615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpti6h0c7i/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8134185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8134575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8135083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8135494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8135893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibid20wz 2022-11-23T02:36:37.8136386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibid20wz/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8136820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:37.8137230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:37.8137653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:37.8138085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:37.8138659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8139266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8139615Z ok (3.967s) 2022-11-23T02:36:37.8139715Z 2022-11-23T02:36:37.8139954Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8140352Z Ran 1 test in 3.968s 2022-11-23T02:36:37.8140500Z 2022-11-23T02:36:37.8140581Z OK 2022-11-23T02:36:37.8140693Z 2022-11-23T02:36:37.8140795Z Generating XML reports... 2022-11-23T02:36:37.8141350Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123023607.xml 2022-11-23T02:36:37.8141977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8142357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8142826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8143229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8143629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgbm0huy 2022-11-23T02:36:37.8144081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgbm0huy/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8144596Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8144822Z 2022-11-23T02:36:37.8144912Z Running tests... 2022-11-23T02:36:37.8145269Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8145688Z test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30718 2022-11-23T02:36:37.8146140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30719 2022-11-23T02:36:37.8146715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8147095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8147587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8147978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8148368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsd8gj6v4 2022-11-23T02:36:37.8148794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsd8gj6v4/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8149340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8149716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8150207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8150604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8151041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpis1k5s_q 2022-11-23T02:36:37.8151491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpis1k5s_q/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8151916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:37.8152296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:37.8152697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:37.8153113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:37.8153675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8154256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8154588Z ok (4.745s) 2022-11-23T02:36:37.8154705Z 2022-11-23T02:36:37.8154940Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8155201Z Ran 1 test in 4.745s 2022-11-23T02:36:37.8155332Z 2022-11-23T02:36:37.8155410Z OK 2022-11-23T02:36:37.8155524Z 2022-11-23T02:36:37.8155626Z Generating XML reports... 2022-11-23T02:36:37.8156428Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123023614.xml 2022-11-23T02:36:37.8157108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8157510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8158032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8158418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8158832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqwi0ixo 2022-11-23T02:36:37.8159314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqwi0ixo/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8159856Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8160090Z 2022-11-23T02:36:37.8160180Z Running tests... 2022-11-23T02:36:37.8160547Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8161026Z test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30909 2022-11-23T02:36:37.8161534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30910 2022-11-23T02:36:37.8162081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8162479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8163001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8163415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8163823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai4z3ztm 2022-11-23T02:36:37.8164290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai4z3ztm/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8164867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8165238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8165746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8166160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8166611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwk9va5n 2022-11-23T02:36:37.8167208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwk9va5n/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8167736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:37.8168129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:37.8168508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:37.8168921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:37.8169479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8170055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8170385Z ok (4.159s) 2022-11-23T02:36:37.8170505Z 2022-11-23T02:36:37.8170737Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8171008Z Ran 1 test in 4.160s 2022-11-23T02:36:37.8171138Z 2022-11-23T02:36:37.8171190Z OK 2022-11-23T02:36:37.8171294Z 2022-11-23T02:36:37.8171392Z Generating XML reports... 2022-11-23T02:36:37.8171942Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123023623.xml 2022-11-23T02:36:37.8172564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8172938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8173417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8173802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8174183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0hsarzyw 2022-11-23T02:36:37.8174608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0hsarzyw/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8175116Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:36:37.8175332Z 2022-11-23T02:36:37.8175418Z Running tests... 2022-11-23T02:36:37.8175766Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8176207Z test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31100 2022-11-23T02:36:37.8176689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31101 2022-11-23T02:36:37.8177197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8177551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8178034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8178420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8178807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwjvffp_ 2022-11-23T02:36:37.8179251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwjvffp_/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8179804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:37.8180175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:37.8180640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:37.8181023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:37.8181442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8ea5xly 2022-11-23T02:36:37.8181891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8ea5xly/_remote_module_non_scriptable.py 2022-11-23T02:36:37.8182312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:37.8182708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:37.8183115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:37.8183680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8184097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:37.8184644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:37.8184965Z ok (4.072s) 2022-11-23T02:36:37.8185083Z 2022-11-23T02:36:37.8185313Z ---------------------------------------------------------------------- 2022-11-23T02:36:37.8185587Z Ran 1 test in 4.073s 2022-11-23T02:36:37.8185722Z 2022-11-23T02:36:37.8185796Z OK 2022-11-23T02:36:37.8185901Z 2022-11-23T02:36:37.8186004Z Generating XML reports... 2022-11-23T02:36:37.8186522Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123023630.xml 2022-11-23T02:36:37.8186838Z 2022-11-23T02:36:37.8187325Z ##[endgroup] 2022-11-23T02:36:37.8187818Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_gloo_ddk8dp0n) 2022-11-23T02:36:37.8188093Z 2022-11-23T02:36:37.8188338Z Running distributed/test_c10d_object_collectives ... [2022-11-23 02:36:37.792000] 2022-11-23T02:36:37.8189009Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:36:37.792934] 2022-11-23T02:36:41.2526126Z 2022-11-23T02:36:41.2527657Z Expand the folded group to see the log file of distributed/test_c10d_object_collectives 2022-11-23T02:36:41.2529827Z ##[group]PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_object_collectives_bde2uk5h) 2022-11-23T02:36:41.2531900Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives 2022-11-23T02:36:41.2532664Z 2022-11-23T02:36:41.2533468Z Running tests... 2022-11-23T02:36:41.2534529Z ---------------------------------------------------------------------- 2022-11-23T02:36:41.2536414Z test_all_gather_object (__main__.TestObjectCollectives) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.485s) 2022-11-23T02:36:41.2538640Z test_broadcast_object_list (__main__.TestObjectCollectives) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:36:41.2540796Z test_gather_object (__main__.TestObjectCollectives) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:36:41.2542945Z test_scatter_object_list (__main__.TestObjectCollectives) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:36:41.2543856Z 2022-11-23T02:36:41.2544538Z ---------------------------------------------------------------------- 2022-11-23T02:36:41.2545317Z Ran 4 tests in 0.488s 2022-11-23T02:36:41.2545690Z 2022-11-23T02:36:41.2545911Z OK (skipped=4) 2022-11-23T02:36:41.2546253Z 2022-11-23T02:36:41.2546512Z Generating XML reports... 2022-11-23T02:36:41.2548237Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221123023639.xml 2022-11-23T02:36:41.2549138Z 2022-11-23T02:36:41.2549846Z ##[endgroup] 2022-11-23T02:36:41.2551421Z FINISHED PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_object_collectives_bde2uk5h) 2022-11-23T02:36:41.2552291Z 2022-11-23T02:36:41.2552943Z Running distributed/test_c10d_gloo ... [2022-11-23 02:36:41.253189] 2022-11-23T02:36:41.2554862Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:36:41.254054] 2022-11-23T02:51:39.3859588Z 2022-11-23T02:51:39.3860625Z Expand the folded group to see the log file of distributed/test_c10d_gloo 2022-11-23T02:51:39.3867194Z ##[group]PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_gloo_eh2eef6g) 2022-11-23T02:51:39.3870020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxjmpdn0 2022-11-23T02:51:39.3872435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxjmpdn0/_remote_module_non_scriptable.py 2022-11-23T02:51:39.3876754Z , <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_rank_membership>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]> 2022-11-23T02:51:39.3880138Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) 2022-11-23T02:51:39.3881087Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) 2022-11-23T02:51:39.3881974Z test_gloo_barrier_device_ids (__main__.CommTest) 2022-11-23T02:51:39.3882848Z test_gloo_rank_membership (__main__.CommTest) 2022-11-23T02:51:39.3883708Z test_gloo_warn_not_in_group (__main__.CommTest) 2022-11-23T02:51:39.3884646Z test_sequence_num_incremented_gloo_default (__main__.CommTest) 2022-11-23T02:51:39.3885641Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) 2022-11-23T02:51:39.3886622Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) 2022-11-23T02:51:39.3888031Z test_sequence_num_set_gloo_new_group (__main__.CommTest) 2022-11-23T02:51:39.3888909Z test_tensor_dtype_complex (__main__.CommTest) 2022-11-23T02:51:39.3889740Z test_tensor_dtype_mismatch (__main__.CommTest) 2022-11-23T02:51:39.3895130Z , <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_cpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_cpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_scatter_work_wait_cpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]> 2022-11-23T02:51:39.3900467Z test_allgather_work_wait_cpu (__main__.CompilerTest) 2022-11-23T02:51:39.3901867Z test_allgather_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:51:39.3903272Z test_allreduce_work_wait_cpu (__main__.CompilerTest) 2022-11-23T02:51:39.3904712Z test_allreduce_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:51:39.3906416Z test_broadcast_work_wait_cpu (__main__.CompilerTest) 2022-11-23T02:51:39.3908359Z test_broadcast_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:51:39.3909882Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) 2022-11-23T02:51:39.3911334Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:51:39.3912786Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) 2022-11-23T02:51:39.3914167Z test_scatter_work_wait_cpu (__main__.CompilerTest) 2022-11-23T02:51:39.3915537Z test_scatter_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:51:39.3940504Z , <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_ignored_sharded_tensor>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]> 2022-11-23T02:51:39.3964496Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3966492Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3968474Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3970564Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3972596Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3974968Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3977022Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3979115Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3981068Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3983033Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3985104Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3987191Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3989295Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3991259Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3993227Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3995174Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3997078Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3998340Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3999006Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.3999720Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4000441Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4001162Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4001939Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4002795Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4003545Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4004291Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4004913Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4005540Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4006232Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4006920Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4007590Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4008239Z test_ignored_sharded_tensor (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4008899Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4009512Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4010134Z test_sparse_gradients (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4010753Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4011513Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4012170Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4013790Z , <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_monitored_barrier>]> 2022-11-23T02:51:39.4015497Z test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T02:51:39.4016281Z test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T02:51:39.4017112Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T02:51:39.4017937Z test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T02:51:39.4018631Z 2022-11-23T02:51:39.4026650Z , <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]> 2022-11-23T02:51:39.4034400Z test_allgather_basics (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4035021Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4035644Z test_allgather_checks (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4036286Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4036890Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4037480Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4038076Z test_allgather_stress (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4038666Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4039419Z test_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4039936Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4040371Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4040831Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4041238Z test_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4041654Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4042080Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4042503Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4042951Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4043391Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4043776Z test_allreduce_stress (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4044182Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4044586Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4044986Z test_broadcast_basics (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4045388Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4045784Z test_broadcast_checks (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4046235Z test_broadcast_stress (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4046635Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4047033Z test_empty_tensors (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4047422Z test_gather_basics (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4047816Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4048205Z test_gather_checks (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4048599Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4049020Z test_gather_stress (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4049421Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4049852Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4050269Z test_reduce_basics (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4050674Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4051044Z test_reduce_checks (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4051422Z test_reduce_stress (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4051807Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4052198Z test_round_robin (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4052666Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4053071Z test_scatter_basics (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4053466Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4053837Z test_scatter_checks (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4054223Z test_scatter_stress (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4054620Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4055025Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4055434Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4055869Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4056275Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-11-23T02:51:39.4057228Z , <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]> 2022-11-23T02:51:39.4058077Z test_forward_backward (__main__.ReducerTest) 2022-11-23T02:51:39.4058456Z test_forward_backward_optimizer (__main__.ReducerTest) 2022-11-23T02:51:39.4058860Z test_forward_backward_unused_parameters (__main__.ReducerTest) 2022-11-23T02:51:39.4059252Z test_multi_dtype_multi_bucket (__main__.ReducerTest) 2022-11-23T02:51:39.4059629Z test_multi_dtype_single_bucket (__main__.ReducerTest) 2022-11-23T02:51:39.4059986Z test_single_dtype_single_bucket (__main__.ReducerTest) 2022-11-23T02:51:39.4060443Z ]> 2022-11-23T02:51:39.4060887Z test_logging_init (__main__.RendezvousEnvTest) 2022-11-23T02:51:39.4061261Z 2022-11-23T02:51:39.4061720Z ]> 2022-11-23T02:51:39.4062178Z test_default_store_timeout_gloo (__main__.TimeoutTest) 2022-11-23T02:51:39.4063118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4063598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4064248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4064767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4065341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpamoyat8i 2022-11-23T02:51:39.4065930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpamoyat8i/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4066600Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4066887Z 2022-11-23T02:51:39.4066995Z Running tests... 2022-11-23T02:51:39.4067438Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4068004Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31407 2022-11-23T02:51:39.4068564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31408 2022-11-23T02:51:39.4069249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4069744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4070381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4070891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4071443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2usi900_ 2022-11-23T02:51:39.4072006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2usi900_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4072726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4073216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4073850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4074363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4074873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4iyn6qg1 2022-11-23T02:51:39.4075455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4iyn6qg1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4075985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4076590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4076962Z ok (3.635s) 2022-11-23T02:51:39.4077122Z 2022-11-23T02:51:39.4077432Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4077785Z Ran 1 test in 3.635s 2022-11-23T02:51:39.4077953Z 2022-11-23T02:51:39.4078046Z OK 2022-11-23T02:51:39.4078181Z 2022-11-23T02:51:39.4078311Z Generating XML reports... 2022-11-23T02:51:39.4078899Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023644.xml 2022-11-23T02:51:39.4079642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4080127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4080770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4081277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4081777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26zht2c3 2022-11-23T02:51:39.4082361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26zht2c3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4082987Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4083269Z 2022-11-23T02:51:39.4083377Z Running tests... 2022-11-23T02:51:39.4083836Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4084467Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31590 2022-11-23T02:51:39.4085032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31591 2022-11-23T02:51:39.4085718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4086203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4086822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4087332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4087836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfgjduhd 2022-11-23T02:51:39.4088430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfgjduhd/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4088990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4089679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4090274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4090921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4091402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4091915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0d35i03z 2022-11-23T02:51:39.4092499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0d35i03z/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4093050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4093581Z ok (4.412s) 2022-11-23T02:51:39.4093736Z 2022-11-23T02:51:39.4094047Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4094379Z Ran 1 test in 4.412s 2022-11-23T02:51:39.4094549Z 2022-11-23T02:51:39.4094642Z OK 2022-11-23T02:51:39.4094779Z 2022-11-23T02:51:39.4094904Z Generating XML reports... 2022-11-23T02:51:39.4095519Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023650.xml 2022-11-23T02:51:39.4096253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4096750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4097388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4097901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4098381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafgb89m8 2022-11-23T02:51:39.4098967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafgb89m8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4099623Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4099905Z 2022-11-23T02:51:39.4100020Z Running tests... 2022-11-23T02:51:39.4100492Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4101046Z test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31777 2022-11-23T02:51:39.4101600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31778 2022-11-23T02:51:39.4102252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4102742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4103438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4103943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4104448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwa4d7lm 2022-11-23T02:51:39.4105040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwa4d7lm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4105591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4106251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4106736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4107373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4107882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4108386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahymo0i0 2022-11-23T02:51:39.4108966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahymo0i0/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4109568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4110084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4110621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4111368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4112133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4112560Z ok (3.634s) 2022-11-23T02:51:39.4112717Z 2022-11-23T02:51:39.4113022Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4113378Z Ran 1 test in 3.635s 2022-11-23T02:51:39.4113548Z 2022-11-23T02:51:39.4113641Z OK 2022-11-23T02:51:39.4113753Z 2022-11-23T02:51:39.4113881Z Generating XML reports... 2022-11-23T02:51:39.4114488Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023658.xml 2022-11-23T02:51:39.4115221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4115713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4116440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4116967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4117470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6r5xpcfo 2022-11-23T02:51:39.4118044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6r5xpcfo/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4118706Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4118985Z 2022-11-23T02:51:39.4119096Z Running tests... 2022-11-23T02:51:39.4119550Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4120093Z test_gloo_rank_membership (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31960 2022-11-23T02:51:39.4120644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31961 2022-11-23T02:51:39.4121313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4121774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4122470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4122982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4123485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfny7ncnl 2022-11-23T02:51:39.4124074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfny7ncnl/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4124627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4125310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4125776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4126412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4126921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4127422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7z9xb20 2022-11-23T02:51:39.4127999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7z9xb20/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4128598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4129131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4129665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4130378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4130942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:51:39.4131663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4132229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:51:39.4132963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:39.4133722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:39.4134140Z ok (3.612s) 2022-11-23T02:51:39.4134294Z 2022-11-23T02:51:39.4134595Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4134929Z Ran 1 test in 3.613s 2022-11-23T02:51:39.4135097Z 2022-11-23T02:51:39.4135189Z OK 2022-11-23T02:51:39.4135324Z 2022-11-23T02:51:39.4135456Z Generating XML reports... 2022-11-23T02:51:39.4136061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023704.xml 2022-11-23T02:51:39.4136796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4137289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4137901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4138411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4138933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn58iusms 2022-11-23T02:51:39.4139521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn58iusms/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4140176Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4140451Z 2022-11-23T02:51:39.4140559Z Running tests... 2022-11-23T02:51:39.4141013Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4141615Z test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32146 2022-11-23T02:51:39.4142141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32147 2022-11-23T02:51:39.4142826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4143314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4143946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4144454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4144961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9h8s9b_ 2022-11-23T02:51:39.4145680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4146142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4146769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4147330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4147838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7aiju09 2022-11-23T02:51:39.4148419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9h8s9b_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4149013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7aiju09/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4149565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4150051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4150582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4151113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4151851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4152618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4153196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:51:39.4153722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:51:39.4154441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:39.4155172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:39.4155596Z ok (4.123s) 2022-11-23T02:51:39.4155749Z 2022-11-23T02:51:39.4159491Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4159859Z Ran 1 test in 4.123s 2022-11-23T02:51:39.4160028Z 2022-11-23T02:51:39.4160121Z OK 2022-11-23T02:51:39.4160265Z 2022-11-23T02:51:39.4160393Z Generating XML reports... 2022-11-23T02:51:39.4161001Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023711.xml 2022-11-23T02:51:39.4161739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4162202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4162839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4163349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4163977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp977vc7kh 2022-11-23T02:51:39.4164562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp977vc7kh/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4165230Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4165512Z 2022-11-23T02:51:39.4165623Z Running tests... 2022-11-23T02:51:39.4166056Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4166624Z test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32332 2022-11-23T02:51:39.4167198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32333 2022-11-23T02:51:39.4167880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4168372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4169008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4169509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4170057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5es8zrnm 2022-11-23T02:51:39.4170641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5es8zrnm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4171196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4171880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4172363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4173001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4173514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4174021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4apphc95 2022-11-23T02:51:39.4174583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4apphc95/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4175142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4175681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4176229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4176970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4177731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4178312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:51:39.4178821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:51:39.4179562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:39.4180327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:39.4180751Z ok (3.914s) 2022-11-23T02:51:39.4180908Z 2022-11-23T02:51:39.4181209Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4181562Z Ran 1 test in 3.914s 2022-11-23T02:51:39.4181729Z 2022-11-23T02:51:39.4181822Z OK 2022-11-23T02:51:39.4181956Z 2022-11-23T02:51:39.4182062Z Generating XML reports... 2022-11-23T02:51:39.4182723Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023718.xml 2022-11-23T02:51:39.4183458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4183952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4184598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4185110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4185615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqotechez 2022-11-23T02:51:39.4186203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqotechez/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4186843Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4187127Z 2022-11-23T02:51:39.4187237Z Running tests... 2022-11-23T02:51:39.4187687Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4188256Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32521 2022-11-23T02:51:39.4188876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32522 2022-11-23T02:51:39.4189555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4190039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4190652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4191159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4191669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwhlab7l 2022-11-23T02:51:39.4192259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwhlab7l/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4192816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4193504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4193991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4194604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4195119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4195626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1470ndke 2022-11-23T02:51:39.4196299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1470ndke/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4196853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4197279Z skip: Need at least 4 CUDA devices (3.418s) 2022-11-23T02:51:39.4197480Z 2022-11-23T02:51:39.4197790Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4198142Z Ran 1 test in 3.419s 2022-11-23T02:51:39.4198293Z 2022-11-23T02:51:39.4198404Z OK (skipped=1) 2022-11-23T02:51:39.4198564Z 2022-11-23T02:51:39.4198689Z Generating XML reports... 2022-11-23T02:51:39.4199296Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023725.xml 2022-11-23T02:51:39.4200025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4200511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4201148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4201719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4202201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl83orfam 2022-11-23T02:51:39.4202787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl83orfam/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4203442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4203719Z 2022-11-23T02:51:39.4203829Z Running tests... 2022-11-23T02:51:39.4204280Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4204841Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32698 2022-11-23T02:51:39.4205400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32699 2022-11-23T02:51:39.4206048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4206538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4207182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4207748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4208256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmc_vcy4 2022-11-23T02:51:39.4208840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmc_vcy4/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4209394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4210066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4210546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4211185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4211697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4212196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4em5s1v3 2022-11-23T02:51:39.4212783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4em5s1v3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4256079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4257428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4258470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4260158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4261676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4262474Z ok (3.718s) 2022-11-23T02:51:39.4262754Z 2022-11-23T02:51:39.4263330Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4263996Z Ran 1 test in 3.719s 2022-11-23T02:51:39.4264306Z 2022-11-23T02:51:39.4264462Z OK 2022-11-23T02:51:39.4264703Z 2022-11-23T02:51:39.4264911Z Generating XML reports... 2022-11-23T02:51:39.4266078Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023731.xml 2022-11-23T02:51:39.4267511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4268435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4269655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4271047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4272000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoip_3bqr 2022-11-23T02:51:39.4273126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoip_3bqr/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4274379Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4274906Z 2022-11-23T02:51:39.4275095Z Running tests... 2022-11-23T02:51:39.4275946Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4277236Z test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32881 2022-11-23T02:51:39.4278312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32882 2022-11-23T02:51:39.4279618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4280542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4281744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4282852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4283805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_o1n6z7o 2022-11-23T02:51:39.4284905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_o1n6z7o/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4286299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4287214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4288437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4289381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4290326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprjn4nrk 2022-11-23T02:51:39.4291440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprjn4nrk/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4292488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4293448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4294454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4295867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4296972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4298355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4299437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:51:39.4300452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:51:39.4301835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:39.4303292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:39.4304074Z ok (3.609s) 2022-11-23T02:51:39.4304350Z 2022-11-23T02:51:39.4304922Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4305554Z Ran 1 test in 3.610s 2022-11-23T02:51:39.4305860Z 2022-11-23T02:51:39.4306017Z OK 2022-11-23T02:51:39.4306367Z 2022-11-23T02:51:39.4306592Z Generating XML reports... 2022-11-23T02:51:39.4307738Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023738.xml 2022-11-23T02:51:39.4309148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4310078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4311298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4312237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4313183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppencm5zd 2022-11-23T02:51:39.4314297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppencm5zd/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4315551Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4316206Z 2022-11-23T02:51:39.4316410Z Running tests... 2022-11-23T02:51:39.4317265Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4318410Z test_tensor_dtype_complex (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33070 2022-11-23T02:51:39.4319461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33071 2022-11-23T02:51:39.4320733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4321649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4322863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4323813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4324763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawgx6l5c 2022-11-23T02:51:39.4325884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawgx6l5c/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4326929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4328230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4329136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4330352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4331292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4332235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf_um_06 2022-11-23T02:51:39.4333329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf_um_06/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4334370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4335357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4336388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4337794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4339259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4340048Z ok (3.724s) 2022-11-23T02:51:39.4340323Z 2022-11-23T02:51:39.4340892Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4341546Z Ran 1 test in 3.725s 2022-11-23T02:51:39.4341849Z 2022-11-23T02:51:39.4342004Z OK 2022-11-23T02:51:39.4342332Z 2022-11-23T02:51:39.4342555Z Generating XML reports... 2022-11-23T02:51:39.4343711Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023745.xml 2022-11-23T02:51:39.4345124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4346046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4347265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4348222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4349174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ttde43_ 2022-11-23T02:51:39.4350261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ttde43_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4351506Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4352038Z 2022-11-23T02:51:39.4352231Z Running tests... 2022-11-23T02:51:39.4353093Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4354198Z test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33253 2022-11-23T02:51:39.4355238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33254 2022-11-23T02:51:39.4356645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4357528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4358745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4359411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4359760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l4z2fqw 2022-11-23T02:51:39.4360162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l4z2fqw/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4360535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4361005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4361327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4361760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4362108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4362453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp617ubeq 2022-11-23T02:51:39.4362855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp617ubeq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4363236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4363601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4363974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4364468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4365348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4366512Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.4367238Z warnings.warn( 2022-11-23T02:51:39.4368194Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.4368830Z warnings.warn( 2022-11-23T02:51:39.4369772Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.4370395Z warnings.warn( 2022-11-23T02:51:39.4371336Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.4371964Z warnings.warn( 2022-11-23T02:51:39.4372180Z ok (3.622s) 2022-11-23T02:51:39.4372315Z 2022-11-23T02:51:39.4372603Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4372929Z Ran 1 test in 3.622s 2022-11-23T02:51:39.4373081Z 2022-11-23T02:51:39.4373223Z OK 2022-11-23T02:51:39.4373345Z 2022-11-23T02:51:39.4373455Z Generating XML reports... 2022-11-23T02:51:39.4374035Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123023751.xml 2022-11-23T02:51:39.4374749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4375204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4375817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4376304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4376776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposxt8xdx 2022-11-23T02:51:39.4377329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposxt8xdx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4377962Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4378225Z 2022-11-23T02:51:39.4378318Z Running tests... 2022-11-23T02:51:39.4378731Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4379266Z test_allgather_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33436 2022-11-23T02:51:39.4379794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33437 2022-11-23T02:51:39.4380431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4381045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4381657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4382144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4382612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0wwnxmb 2022-11-23T02:51:39.4383163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0wwnxmb/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4383862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4384330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4384944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4385495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4385965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6z82eplo 2022-11-23T02:51:39.4386522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6z82eplo/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4387049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4387529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4388029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4388749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4389305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4390003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4391060Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4391837Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4392757Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4393514Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4393841Z ok (3.514s) 2022-11-23T02:51:39.4393989Z 2022-11-23T02:51:39.4394272Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4394605Z Ran 1 test in 3.515s 2022-11-23T02:51:39.4394764Z 2022-11-23T02:51:39.4394840Z OK 2022-11-23T02:51:39.4394958Z 2022-11-23T02:51:39.4395077Z Generating XML reports... 2022-11-23T02:51:39.4395661Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023758.xml 2022-11-23T02:51:39.4396472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4396934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4397545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4398025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4398503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbk13al__ 2022-11-23T02:51:39.4399060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbk13al__/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4399693Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4399958Z 2022-11-23T02:51:39.4400044Z Running tests... 2022-11-23T02:51:39.4400474Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4401004Z test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33619 2022-11-23T02:51:39.4401539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33620 2022-11-23T02:51:39.4402189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4402646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4403258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4403792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4404270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl23zqeeb 2022-11-23T02:51:39.4404837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl23zqeeb/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4405371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4406038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4406496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4407110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4407591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4408062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbclnbnub 2022-11-23T02:51:39.4408618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbclnbnub/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4409142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4409710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4410234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4410944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4411677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4412670Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4413433Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4414364Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4415139Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4415470Z ok (4.027s) 2022-11-23T02:51:39.4415608Z 2022-11-23T02:51:39.4415904Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4416241Z Ran 1 test in 4.028s 2022-11-23T02:51:39.4416399Z 2022-11-23T02:51:39.4416485Z OK 2022-11-23T02:51:39.4416602Z 2022-11-23T02:51:39.4416725Z Generating XML reports... 2022-11-23T02:51:39.4417304Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023804.xml 2022-11-23T02:51:39.4418039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4418499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4419117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4419611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4420092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w_72h6_ 2022-11-23T02:51:39.4420650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w_72h6_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4421271Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4421582Z 2022-11-23T02:51:39.4421687Z Running tests... 2022-11-23T02:51:39.4422118Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4422661Z test_allreduce_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33806 2022-11-23T02:51:39.4423202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33807 2022-11-23T02:51:39.4423843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4424299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4424897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4425378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4425872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tdr26am 2022-11-23T02:51:39.4426430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tdr26am/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4426953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4427655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4428112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4428723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4429201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4429685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7skatnlr 2022-11-23T02:51:39.4430229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7skatnlr/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4430748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4431242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4431751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4432445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4433180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4434172Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4434950Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4435887Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4436742Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4437654Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4438403Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4439313Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4440131Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4440447Z ok (3.714s) 2022-11-23T02:51:39.4440586Z 2022-11-23T02:51:39.4440884Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4441203Z Ran 1 test in 3.714s 2022-11-23T02:51:39.4441363Z 2022-11-23T02:51:39.4441452Z OK 2022-11-23T02:51:39.4441571Z 2022-11-23T02:51:39.4441687Z Generating XML reports... 2022-11-23T02:51:39.4442285Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023811.xml 2022-11-23T02:51:39.4443013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4443472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4444080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4444567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4445059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp920zm17q 2022-11-23T02:51:39.4445677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp920zm17q/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4446316Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4446586Z 2022-11-23T02:51:39.4446686Z Running tests... 2022-11-23T02:51:39.4447127Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4447671Z test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33989 2022-11-23T02:51:39.4448207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33990 2022-11-23T02:51:39.4448870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4449339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4449967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4450456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4450944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwdpc4al 2022-11-23T02:51:39.4451514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwdpc4al/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4452035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4452704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4453171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4453792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4454279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4454771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpmc8qmp 2022-11-23T02:51:39.4455341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpmc8qmp/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4455860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4456372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4456893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4457603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4458403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4459414Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4460190Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4461122Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4461891Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4462820Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4463622Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4464538Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4465298Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4465633Z ok (3.909s) 2022-11-23T02:51:39.4465773Z 2022-11-23T02:51:39.4466062Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4466406Z Ran 1 test in 3.909s 2022-11-23T02:51:39.4466570Z 2022-11-23T02:51:39.4466655Z OK 2022-11-23T02:51:39.4466781Z 2022-11-23T02:51:39.4466900Z Generating XML reports... 2022-11-23T02:51:39.4467489Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023818.xml 2022-11-23T02:51:39.4468225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4468696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4469315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4469807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4470292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xgszmkp 2022-11-23T02:51:39.4470858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xgszmkp/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4471489Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4471758Z 2022-11-23T02:51:39.4471858Z Running tests... 2022-11-23T02:51:39.4472298Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4472835Z test_broadcast_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34176 2022-11-23T02:51:39.4473387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34177 2022-11-23T02:51:39.4474044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4474514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4475133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4475607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4476248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgv62wch 2022-11-23T02:51:39.4476822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgv62wch/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4477536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4478008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4478630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4479117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4479591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp000fh8um 2022-11-23T02:51:39.4480156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp000fh8um/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4480701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4481201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4481718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4482305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4483030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4483771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4484770Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4485712Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4486658Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4487427Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4487765Z ok (3.723s) 2022-11-23T02:51:39.4487910Z 2022-11-23T02:51:39.4488202Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4488537Z Ran 1 test in 3.724s 2022-11-23T02:51:39.4488695Z 2022-11-23T02:51:39.4488779Z OK 2022-11-23T02:51:39.4488906Z 2022-11-23T02:51:39.4489009Z Generating XML reports... 2022-11-23T02:51:39.4489612Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023825.xml 2022-11-23T02:51:39.4490353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4490826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4491452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4491948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4492440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2957udu8 2022-11-23T02:51:39.4492993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2957udu8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4493635Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4493907Z 2022-11-23T02:51:39.4494012Z Running tests... 2022-11-23T02:51:39.4494455Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4495057Z test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34359 2022-11-23T02:51:39.4495601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34360 2022-11-23T02:51:39.4496269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4496744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4497354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4497847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4498334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxj38qicc 2022-11-23T02:51:39.4498907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxj38qicc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4499447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4500117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4500646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4501254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4501748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4502233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_d6xjwc 2022-11-23T02:51:39.4502796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_d6xjwc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4503330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4503849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4504380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4505079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4505833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4506855Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4507632Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4508563Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4509337Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4509673Z ok (3.918s) 2022-11-23T02:51:39.4509822Z 2022-11-23T02:51:39.4510116Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4510457Z Ran 1 test in 3.919s 2022-11-23T02:51:39.4510618Z 2022-11-23T02:51:39.4510687Z OK 2022-11-23T02:51:39.4510816Z 2022-11-23T02:51:39.4510934Z Generating XML reports... 2022-11-23T02:51:39.4511543Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023831.xml 2022-11-23T02:51:39.4512274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4512740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4513418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4513915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4514391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a4uizko 2022-11-23T02:51:39.4514958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a4uizko/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4515609Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4515881Z 2022-11-23T02:51:39.4515985Z Running tests... 2022-11-23T02:51:39.4516511Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4517069Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34546 2022-11-23T02:51:39.4517631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34547 2022-11-23T02:51:39.4518288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4518740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4519448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4519947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4520434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2z2r88fw 2022-11-23T02:51:39.4521002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2z2r88fw/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4521534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4522207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4522669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4523289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4523782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4524264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hii3cmm 2022-11-23T02:51:39.4524833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hii3cmm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4525371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4525889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4526399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4527124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4527879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4528891Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4529668Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4530604Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4531437Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4532364Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4533130Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4534055Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4534812Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4535718Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4536484Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4537450Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4538212Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4539136Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4539895Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4540824Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4541580Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4541919Z ok (3.613s) 2022-11-23T02:51:39.4542062Z 2022-11-23T02:51:39.4542359Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4542687Z Ran 1 test in 3.614s 2022-11-23T02:51:39.4542846Z 2022-11-23T02:51:39.4542933Z OK 2022-11-23T02:51:39.4543063Z 2022-11-23T02:51:39.4543185Z Generating XML reports... 2022-11-23T02:51:39.4543787Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023838.xml 2022-11-23T02:51:39.4544520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4544991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4545614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4546096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4546587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwv4w5na 2022-11-23T02:51:39.4547159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwv4w5na/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4547803Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4548073Z 2022-11-23T02:51:39.4548174Z Running tests... 2022-11-23T02:51:39.4548613Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4549165Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34729 2022-11-23T02:51:39.4549779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34730 2022-11-23T02:51:39.4550422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4550896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4551514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4552005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4552492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl_vg7md 2022-11-23T02:51:39.4553061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl_vg7md/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4553601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4554255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4554726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4555406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4555891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4556460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v3srpzx 2022-11-23T02:51:39.4557033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v3srpzx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4557568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4558073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4558604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4559335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4560086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4561102Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4561879Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4562817Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4563592Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4564524Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4565285Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4566215Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4566961Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4567885Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4568715Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4569652Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4570418Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4571340Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4572110Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4573080Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4573839Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4574173Z ok (3.914s) 2022-11-23T02:51:39.4574320Z 2022-11-23T02:51:39.4574612Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4574937Z Ran 1 test in 3.914s 2022-11-23T02:51:39.4575097Z 2022-11-23T02:51:39.4575184Z OK 2022-11-23T02:51:39.4575314Z 2022-11-23T02:51:39.4575433Z Generating XML reports... 2022-11-23T02:51:39.4576048Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023845.xml 2022-11-23T02:51:39.4576787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4577261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4577892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4578370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4578866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6euhbvs 2022-11-23T02:51:39.4579449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6euhbvs/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4580097Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4580369Z 2022-11-23T02:51:39.4580475Z Running tests... 2022-11-23T02:51:39.4580921Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4581479Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34916 2022-11-23T02:51:39.4582022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34917 2022-11-23T02:51:39.4582675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4583128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4583732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4584201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4584685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbqr3ocm 2022-11-23T02:51:39.4585261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbqr3ocm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4585989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4586663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4587133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4587765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4588259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4588746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8scy4i56 2022-11-23T02:51:39.4589311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8scy4i56/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4589846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4590347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4591070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4591640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4592407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4593421Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4594194Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4595130Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4595895Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4596940Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4597708Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4598620Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4599382Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4599732Z ok (3.631s) 2022-11-23T02:51:39.4599879Z 2022-11-23T02:51:39.4600173Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4600516Z Ran 1 test in 3.632s 2022-11-23T02:51:39.4600678Z 2022-11-23T02:51:39.4600797Z OK 2022-11-23T02:51:39.4600931Z 2022-11-23T02:51:39.4601056Z Generating XML reports... 2022-11-23T02:51:39.4601645Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023852.xml 2022-11-23T02:51:39.4602373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4602856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4603484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4603978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4604541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79v9e0xe 2022-11-23T02:51:39.4605109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79v9e0xe/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4605739Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4606012Z 2022-11-23T02:51:39.4606115Z Running tests... 2022-11-23T02:51:39.4606558Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4607099Z test_scatter_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35099 2022-11-23T02:51:39.4607646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35100 2022-11-23T02:51:39.4608303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4608772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4609398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4609872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4610424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphdg4j0fb 2022-11-23T02:51:39.4610998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphdg4j0fb/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4611701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4612176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4612797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4613293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4613769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9abg6oro 2022-11-23T02:51:39.4614335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9abg6oro/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4614878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4615378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4615892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4616608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4617168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4617874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4618878Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4619653Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4620588Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4621360Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4621701Z ok (3.632s) 2022-11-23T02:51:39.4621848Z 2022-11-23T02:51:39.4622140Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4622538Z Ran 1 test in 3.632s 2022-11-23T02:51:39.4622700Z 2022-11-23T02:51:39.4622786Z OK 2022-11-23T02:51:39.4622916Z 2022-11-23T02:51:39.4623020Z Generating XML reports... 2022-11-23T02:51:39.4623628Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023858.xml 2022-11-23T02:51:39.4624367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4624836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4625461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4625957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4626446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02w6ztq_ 2022-11-23T02:51:39.4627005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02w6ztq_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4627646Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4627920Z 2022-11-23T02:51:39.4628028Z Running tests... 2022-11-23T02:51:39.4628467Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4629056Z test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35282 2022-11-23T02:51:39.4629602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35283 2022-11-23T02:51:39.4630264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4630734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4631338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4631835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4632329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfu2yxaiq 2022-11-23T02:51:39.4632898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfu2yxaiq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4633436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4634109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4634581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4635190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4635691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4636273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoieswxze 2022-11-23T02:51:39.4636853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoieswxze/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4637389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4637912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4638442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4639146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4639906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4640921Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4641775Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4642721Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:51:39.4643497Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:51:39.4643832Z ok (4.014s) 2022-11-23T02:51:39.4643975Z 2022-11-23T02:51:39.4644265Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4644604Z Ran 1 test in 4.015s 2022-11-23T02:51:39.4644762Z 2022-11-23T02:51:39.4644831Z OK 2022-11-23T02:51:39.4644957Z 2022-11-23T02:51:39.4645072Z Generating XML reports... 2022-11-23T02:51:39.4645673Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123023905.xml 2022-11-23T02:51:39.4646405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4646873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4647545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4648037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4648510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvpm69bc 2022-11-23T02:51:39.4649078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvpm69bc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4649728Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4649997Z 2022-11-23T02:51:39.4650097Z Running tests... 2022-11-23T02:51:39.4650536Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4651003Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4651814Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35469 2022-11-23T02:51:39.4652396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35470 2022-11-23T02:51:39.4653035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4653503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4654127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4654620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4655109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz064o43h 2022-11-23T02:51:39.4655674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz064o43h/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4656206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4656858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4657330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4657948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4658437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4658921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsm2zzgj4 2022-11-23T02:51:39.4659485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsm2zzgj4/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4660093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4660436Z ok (6.221s) 2022-11-23T02:51:39.4660458Z 2022-11-23T02:51:39.4660734Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4660843Z Ran 1 test in 6.222s 2022-11-23T02:51:39.4660850Z 2022-11-23T02:51:39.4660934Z OK 2022-11-23T02:51:39.4660941Z 2022-11-23T02:51:39.4661058Z Generating XML reports... 2022-11-23T02:51:39.4661573Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123023912.xml 2022-11-23T02:51:39.4661976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4662152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4662562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4662761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4663021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaimp0dyr 2022-11-23T02:51:39.4663346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaimp0dyr/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4663684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4663691Z 2022-11-23T02:51:39.4663791Z Running tests... 2022-11-23T02:51:39.4664077Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4664323Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4664658Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35662 2022-11-23T02:51:39.4664886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35663 2022-11-23T02:51:39.4665292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4665467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4665883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4666074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4666322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu92uvhrc 2022-11-23T02:51:39.4666599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu92uvhrc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4666832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4667231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4667410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4667821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4668015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4668274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tfr__0y 2022-11-23T02:51:39.4668547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tfr__0y/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4668778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4668871Z ok (6.323s) 2022-11-23T02:51:39.4668878Z 2022-11-23T02:51:39.4669166Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4669271Z Ran 1 test in 6.324s 2022-11-23T02:51:39.4669322Z 2022-11-23T02:51:39.4669407Z OK 2022-11-23T02:51:39.4669413Z 2022-11-23T02:51:39.4669532Z Generating XML reports... 2022-11-23T02:51:39.4670039Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123023921.xml 2022-11-23T02:51:39.4670443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4670619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4671028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4671218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4671475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrvnzwjx 2022-11-23T02:51:39.4671736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrvnzwjx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4672073Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4672080Z 2022-11-23T02:51:39.4672182Z Running tests... 2022-11-23T02:51:39.4672467Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4672763Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4673092Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35855 2022-11-23T02:51:39.4673314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35856 2022-11-23T02:51:39.4673715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4673893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4674304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4674497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4674756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwmeaf2s 2022-11-23T02:51:39.4675034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwmeaf2s/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4675267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4675666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4675840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4676373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4676565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4676828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0coh4cek 2022-11-23T02:51:39.4677103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0coh4cek/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4677337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4677581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4677807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4678042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4678277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4679308Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4679485Z warnings.warn( 2022-11-23T02:51:39.4679726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4680746Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4680857Z warnings.warn( 2022-11-23T02:51:39.4681251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4681480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4681714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4681946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4682227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4690732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4691037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4691131Z ok (6.524s) 2022-11-23T02:51:39.4691139Z 2022-11-23T02:51:39.4691486Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4691588Z Ran 1 test in 6.525s 2022-11-23T02:51:39.4691595Z 2022-11-23T02:51:39.4691676Z OK 2022-11-23T02:51:39.4691684Z 2022-11-23T02:51:39.4691798Z Generating XML reports... 2022-11-23T02:51:39.4692331Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123023930.xml 2022-11-23T02:51:39.4692741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4692920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4693336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4693529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4693779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzv_bm98x 2022-11-23T02:51:39.4694055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzv_bm98x/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4694393Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4694404Z 2022-11-23T02:51:39.4694499Z Running tests... 2022-11-23T02:51:39.4694782Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4695028Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4695356Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36048 2022-11-23T02:51:39.4695575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36049 2022-11-23T02:51:39.4695978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4696149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4696559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4696746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4697156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0hymd_h 2022-11-23T02:51:39.4697435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0hymd_h/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4697668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4698068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4698240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4698649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4698835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4699091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyi82ldiq 2022-11-23T02:51:39.4699371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyi82ldiq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4699601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4699826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4700109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4700347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4700578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4701599Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4701705Z warnings.warn( 2022-11-23T02:51:39.4701935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4702946Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4703046Z warnings.warn( 2022-11-23T02:51:39.4703277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4703505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4703735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4703972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4704199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4704424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4704652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4704740Z ok (6.527s) 2022-11-23T02:51:39.4704748Z 2022-11-23T02:51:39.4705035Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4705135Z Ran 1 test in 6.528s 2022-11-23T02:51:39.4705143Z 2022-11-23T02:51:39.4705220Z OK 2022-11-23T02:51:39.4705226Z 2022-11-23T02:51:39.4705338Z Generating XML reports... 2022-11-23T02:51:39.4705842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123023940.xml 2022-11-23T02:51:39.4706291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4706463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4706875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4707054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4707309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0016ic0 2022-11-23T02:51:39.4707584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0016ic0/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4707920Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4707928Z 2022-11-23T02:51:39.4708026Z Running tests... 2022-11-23T02:51:39.4708313Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4708587Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4709075Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36241 2022-11-23T02:51:39.4709342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36242 2022-11-23T02:51:39.4709743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4709917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4710323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4710509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4710772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6_qzrxe 2022-11-23T02:51:39.4711049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6_qzrxe/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4711278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4711675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4711846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4712260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4712448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4712706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wk88zzs 2022-11-23T02:51:39.4712980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wk88zzs/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4713203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4713439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4713670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4713903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4714130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4714222Z ok (6.524s) 2022-11-23T02:51:39.4714229Z 2022-11-23T02:51:39.4714515Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4714617Z Ran 1 test in 6.525s 2022-11-23T02:51:39.4714624Z 2022-11-23T02:51:39.4714704Z OK 2022-11-23T02:51:39.4714710Z 2022-11-23T02:51:39.4714821Z Generating XML reports... 2022-11-23T02:51:39.4715326Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123023949.xml 2022-11-23T02:51:39.4715784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4715954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4716487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4716674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4716931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ektgwmx 2022-11-23T02:51:39.4717201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ektgwmx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4717536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4717544Z 2022-11-23T02:51:39.4717641Z Running tests... 2022-11-23T02:51:39.4717932Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4718197Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4718663Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36434 2022-11-23T02:51:39.4718944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36435 2022-11-23T02:51:39.4719347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4719519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4719928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4720115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4720372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthoybchd 2022-11-23T02:51:39.4720650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthoybchd/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4720879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4721277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4721448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4721858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4722044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4722300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1tnwvcv 2022-11-23T02:51:39.4722571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1tnwvcv/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4722799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4723039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4723275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4723498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4723730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4723818Z ok (6.223s) 2022-11-23T02:51:39.4723825Z 2022-11-23T02:51:39.4724098Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4724199Z Ran 1 test in 6.224s 2022-11-23T02:51:39.4724206Z 2022-11-23T02:51:39.4724283Z OK 2022-11-23T02:51:39.4724290Z 2022-11-23T02:51:39.4724402Z Generating XML reports... 2022-11-23T02:51:39.4724899Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123023959.xml 2022-11-23T02:51:39.4725351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4725522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4725933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4726121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4726378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9lwe4hx 2022-11-23T02:51:39.4726655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9lwe4hx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4726982Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4726989Z 2022-11-23T02:51:39.4727088Z Running tests... 2022-11-23T02:51:39.4727371Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4727618Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4728173Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36627 2022-11-23T02:51:39.4728393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36628 2022-11-23T02:51:39.4728795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4728965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4729373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4729561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4729823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqirtta6z 2022-11-23T02:51:39.4730088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqirtta6z/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4730318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4730717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4730889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4731293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4731479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4731729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcejdce5 2022-11-23T02:51:39.4732000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcejdce5/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4732230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4732463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4732698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4733524Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:51:39.4734398Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:51:39.4734631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4734867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4734959Z ok (6.924s) 2022-11-23T02:51:39.4734966Z 2022-11-23T02:51:39.4735254Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4735359Z Ran 1 test in 6.925s 2022-11-23T02:51:39.4735368Z 2022-11-23T02:51:39.4735449Z OK 2022-11-23T02:51:39.4735456Z 2022-11-23T02:51:39.4735569Z Generating XML reports... 2022-11-23T02:51:39.4736072Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024008.xml 2022-11-23T02:51:39.4736536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4736709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4737123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4737309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4737565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdfv6bso3 2022-11-23T02:51:39.4737839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdfv6bso3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4738171Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4738178Z 2022-11-23T02:51:39.4738277Z Running tests... 2022-11-23T02:51:39.4738566Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4738801Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4739305Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36820 2022-11-23T02:51:39.4739532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36821 2022-11-23T02:51:39.4739929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4740095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4740506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4740695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4740951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot_f1nru 2022-11-23T02:51:39.4741226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot_f1nru/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4741451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4741844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4742011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4742415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4742600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4742903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr55ko77f 2022-11-23T02:51:39.4743172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr55ko77f/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4743397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4743489Z ok (6.027s) 2022-11-23T02:51:39.4743497Z 2022-11-23T02:51:39.4743778Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4743877Z Ran 1 test in 6.028s 2022-11-23T02:51:39.4743884Z 2022-11-23T02:51:39.4743960Z OK 2022-11-23T02:51:39.4743966Z 2022-11-23T02:51:39.4744077Z Generating XML reports... 2022-11-23T02:51:39.4744570Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024018.xml 2022-11-23T02:51:39.4744963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4745133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4745536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4745723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4746024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvoae2j7o 2022-11-23T02:51:39.4746299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvoae2j7o/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4746631Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4746638Z 2022-11-23T02:51:39.4746735Z Running tests... 2022-11-23T02:51:39.4747009Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4747250Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4747592Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37013 2022-11-23T02:51:39.4747813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37014 2022-11-23T02:51:39.4748217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4748390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4748797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4748988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4749246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvogrh0fs 2022-11-23T02:51:39.4749521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvogrh0fs/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4749752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4750148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4750319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4750716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4750904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4751159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfej2kdjw 2022-11-23T02:51:39.4751430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfej2kdjw/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4751658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4751890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4752168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4752396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4752631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4752720Z ok (6.233s) 2022-11-23T02:51:39.4752727Z 2022-11-23T02:51:39.4753013Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4753112Z Ran 1 test in 6.233s 2022-11-23T02:51:39.4753119Z 2022-11-23T02:51:39.4753194Z OK 2022-11-23T02:51:39.4753200Z 2022-11-23T02:51:39.4753310Z Generating XML reports... 2022-11-23T02:51:39.4753814Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024027.xml 2022-11-23T02:51:39.4754210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4754382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4754790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4755018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4755277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi89kdxyy 2022-11-23T02:51:39.4755551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi89kdxyy/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4755877Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4755893Z 2022-11-23T02:51:39.4755978Z Running tests... 2022-11-23T02:51:39.4756360Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4756622Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4756958Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37206 2022-11-23T02:51:39.4757174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37207 2022-11-23T02:51:39.4757576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4757748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4758152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4758340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4758596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2hjv2wn 2022-11-23T02:51:39.4758864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2hjv2wn/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4759096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4759491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4759663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4760064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4760246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4760499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwwe3eq2 2022-11-23T02:51:39.4760770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwwe3eq2/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4760997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4761887Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:51:39.4762710Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:51:39.4763780Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4763884Z warnings.warn( 2022-11-23T02:51:39.4764120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4765125Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4765226Z warnings.warn( 2022-11-23T02:51:39.4765460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4765683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4765913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4766000Z ok (7.017s) 2022-11-23T02:51:39.4766008Z 2022-11-23T02:51:39.4766292Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4766390Z Ran 1 test in 7.018s 2022-11-23T02:51:39.4766397Z 2022-11-23T02:51:39.4766471Z OK 2022-11-23T02:51:39.4766477Z 2022-11-23T02:51:39.4766587Z Generating XML reports... 2022-11-23T02:51:39.4767077Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024036.xml 2022-11-23T02:51:39.4767480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4767653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4768062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4768249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4768504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldp_rz6r 2022-11-23T02:51:39.4768778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldp_rz6r/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4769105Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4769112Z 2022-11-23T02:51:39.4769205Z Running tests... 2022-11-23T02:51:39.4769485Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4769788Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4770119Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37399 2022-11-23T02:51:39.4770338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37400 2022-11-23T02:51:39.4770739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4770914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4771320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4771507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4771765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoim969vv 2022-11-23T02:51:39.4772038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoim969vv/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4772264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4772704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4772880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4773287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4773464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4773716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0kn1pry 2022-11-23T02:51:39.4773986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0kn1pry/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4774212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4775221Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4775322Z warnings.warn( 2022-11-23T02:51:39.4775556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4776561Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4776664Z warnings.warn( 2022-11-23T02:51:39.4776902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4777130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4777358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4777449Z ok (7.024s) 2022-11-23T02:51:39.4777456Z 2022-11-23T02:51:39.4777739Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4777843Z Ran 1 test in 7.025s 2022-11-23T02:51:39.4777850Z 2022-11-23T02:51:39.4777925Z OK 2022-11-23T02:51:39.4777931Z 2022-11-23T02:51:39.4778042Z Generating XML reports... 2022-11-23T02:51:39.4778540Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024046.xml 2022-11-23T02:51:39.4778937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4779158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4779568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4779756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4780008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6szsdg2 2022-11-23T02:51:39.4780281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6szsdg2/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4780612Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4780619Z 2022-11-23T02:51:39.4780711Z Running tests... 2022-11-23T02:51:39.4780985Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4781410Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4781710Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37592 2022-11-23T02:51:39.4781923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37593 2022-11-23T02:51:39.4782367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4782541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4782947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4783131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4783386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsdpf0pe 2022-11-23T02:51:39.4783656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsdpf0pe/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4783885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4784279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4784450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4784854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4785038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4785289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcyc8x2i 2022-11-23T02:51:39.4785559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcyc8x2i/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4785783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4786020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4786253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4786489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4786708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4786795Z ok (6.219s) 2022-11-23T02:51:39.4786802Z 2022-11-23T02:51:39.4787084Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4787181Z Ran 1 test in 6.220s 2022-11-23T02:51:39.4787187Z 2022-11-23T02:51:39.4787266Z OK 2022-11-23T02:51:39.4787272Z 2022-11-23T02:51:39.4787383Z Generating XML reports... 2022-11-23T02:51:39.4787891Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024056.xml 2022-11-23T02:51:39.4788340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4788516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4788924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4789110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4789369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwj3qc8yf 2022-11-23T02:51:39.4789643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwj3qc8yf/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4789971Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4789978Z 2022-11-23T02:51:39.4790071Z Running tests... 2022-11-23T02:51:39.4790352Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4790615Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4790917Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37785 2022-11-23T02:51:39.4791188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37786 2022-11-23T02:51:39.4791591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4791760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4792161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4792348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4792604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsg9w1z3d 2022-11-23T02:51:39.4792885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsg9w1z3d/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4793110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4793511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4793681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4794089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4794277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4794531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8oh18nmr 2022-11-23T02:51:39.4794798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8oh18nmr/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4795028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4795260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4795492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4795724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4795947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4796282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4796508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4796729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4796954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4797106Z ok (6.837s) 2022-11-23T02:51:39.4797113Z 2022-11-23T02:51:39.4797394Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4797495Z Ran 1 test in 6.838s 2022-11-23T02:51:39.4797501Z 2022-11-23T02:51:39.4797579Z OK 2022-11-23T02:51:39.4797586Z 2022-11-23T02:51:39.4797694Z Generating XML reports... 2022-11-23T02:51:39.4798210Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024105.xml 2022-11-23T02:51:39.4798606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4798776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4799185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4799372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4799636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6ch369y 2022-11-23T02:51:39.4799907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6ch369y/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4800236Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4800292Z 2022-11-23T02:51:39.4800392Z Running tests... 2022-11-23T02:51:39.4800677Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4800898Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4801233Z This unit test verifies whether the Future object is passed properly. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37978 2022-11-23T02:51:39.4801450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37979 2022-11-23T02:51:39.4801846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4802025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4802432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4802621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4802880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1hopv9e2 2022-11-23T02:51:39.4803141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1hopv9e2/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4803367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4803764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4803935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4804345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4804534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4804789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafpwahpt 2022-11-23T02:51:39.4805066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafpwahpt/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4805292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4805381Z ok (3.618s) 2022-11-23T02:51:39.4805388Z 2022-11-23T02:51:39.4805670Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4805772Z Ran 1 test in 3.619s 2022-11-23T02:51:39.4805779Z 2022-11-23T02:51:39.4805855Z OK 2022-11-23T02:51:39.4805861Z 2022-11-23T02:51:39.4805971Z Generating XML reports... 2022-11-23T02:51:39.4806479Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024115.xml 2022-11-23T02:51:39.4806940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4807110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4807524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4807707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4807965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzdp1yrn 2022-11-23T02:51:39.4808240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzdp1yrn/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4808562Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4808581Z 2022-11-23T02:51:39.4808671Z Running tests... 2022-11-23T02:51:39.4808953Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4809179Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4809578Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38165 2022-11-23T02:51:39.4809798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38166 2022-11-23T02:51:39.4810195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4810361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4810767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4810952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4811209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp24_04w 2022-11-23T02:51:39.4811476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp24_04w/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4811701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4812095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4812262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4812675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4812857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4813110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zlgz_vi 2022-11-23T02:51:39.4813378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zlgz_vi/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4813607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4813695Z ok (4.030s) 2022-11-23T02:51:39.4813702Z 2022-11-23T02:51:39.4813988Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4814078Z Ran 1 test in 4.031s 2022-11-23T02:51:39.4814093Z 2022-11-23T02:51:39.4814163Z OK 2022-11-23T02:51:39.4814168Z 2022-11-23T02:51:39.4814281Z Generating XML reports... 2022-11-23T02:51:39.4814786Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024122.xml 2022-11-23T02:51:39.4815181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4815353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4815760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4815990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4816248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdvzhpdm 2022-11-23T02:51:39.4816523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdvzhpdm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4816854Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4816861Z 2022-11-23T02:51:39.4816954Z Running tests... 2022-11-23T02:51:39.4817234Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4817456Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4817802Z DDP communication hook can only be registered once. This test validates whether ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38356 2022-11-23T02:51:39.4818023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38357 2022-11-23T02:51:39.4818423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4818590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4819046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4819235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4819493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzaafa32b 2022-11-23T02:51:39.4819766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzaafa32b/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4819986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4820379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4820555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4820958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4821145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4821400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fd8pouq 2022-11-23T02:51:39.4821676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fd8pouq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4821904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4821997Z ok (3.719s) 2022-11-23T02:51:39.4822004Z 2022-11-23T02:51:39.4822286Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4822391Z Ran 1 test in 3.719s 2022-11-23T02:51:39.4822398Z 2022-11-23T02:51:39.4822474Z OK 2022-11-23T02:51:39.4822480Z 2022-11-23T02:51:39.4822591Z Generating XML reports... 2022-11-23T02:51:39.4823094Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024129.xml 2022-11-23T02:51:39.4823486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4823656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4824066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4824252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4824507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vjiqgvm 2022-11-23T02:51:39.4824787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vjiqgvm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4825174Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4825181Z 2022-11-23T02:51:39.4825266Z Running tests... 2022-11-23T02:51:39.4825550Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4825775Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4826118Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38539 2022-11-23T02:51:39.4826336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38540 2022-11-23T02:51:39.4826734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4826905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4827315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4827511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4827771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2ajbu3d 2022-11-23T02:51:39.4828091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2ajbu3d/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4828318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4828722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4828895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4829305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4829492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4829752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0uiqndzv 2022-11-23T02:51:39.4830023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0uiqndzv/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4830254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4830344Z ok (3.735s) 2022-11-23T02:51:39.4830351Z 2022-11-23T02:51:39.4830641Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4830731Z Ran 1 test in 3.735s 2022-11-23T02:51:39.4830750Z 2022-11-23T02:51:39.4830820Z OK 2022-11-23T02:51:39.4830827Z 2022-11-23T02:51:39.4830942Z Generating XML reports... 2022-11-23T02:51:39.4831446Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024135.xml 2022-11-23T02:51:39.4831845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4832022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4832439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4832630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4832889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8wm3hnm 2022-11-23T02:51:39.4833160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8wm3hnm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4833490Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4833497Z 2022-11-23T02:51:39.4833594Z Running tests... 2022-11-23T02:51:39.4833876Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4834092Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4834488Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38780 2022-11-23T02:51:39.4834711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38781 2022-11-23T02:51:39.4835117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4835291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4835699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4835889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4836238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjsg71hxp 2022-11-23T02:51:39.4836515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjsg71hxp/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4836736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4837140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4837374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4837787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4837971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4838225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnuz0oaz 2022-11-23T02:51:39.4838496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnuz0oaz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4838724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4838816Z ok (3.615s) 2022-11-23T02:51:39.4838824Z 2022-11-23T02:51:39.4839109Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4839207Z Ran 1 test in 3.615s 2022-11-23T02:51:39.4839213Z 2022-11-23T02:51:39.4839292Z OK 2022-11-23T02:51:39.4839299Z 2022-11-23T02:51:39.4839411Z Generating XML reports... 2022-11-23T02:51:39.4839915Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024142.xml 2022-11-23T02:51:39.4840320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4840490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4840904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4841093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4841351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_vaa5_3 2022-11-23T02:51:39.4841622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_vaa5_3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4841952Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4841961Z 2022-11-23T02:51:39.4842059Z Running tests... 2022-11-23T02:51:39.4842331Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4842556Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4842904Z This test checks whether return annotation checked properly if defined. It also ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38963 2022-11-23T02:51:39.4843124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38964 2022-11-23T02:51:39.4843524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4843746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4844156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4844346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4844604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3673njbz 2022-11-23T02:51:39.4844877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3673njbz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4845109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4845510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4845680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4846091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4846276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4846533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fyltcbi 2022-11-23T02:51:39.4846847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fyltcbi/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4847078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4847169Z ok (3.735s) 2022-11-23T02:51:39.4847176Z 2022-11-23T02:51:39.4847468Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4847574Z Ran 1 test in 3.735s 2022-11-23T02:51:39.4847580Z 2022-11-23T02:51:39.4847649Z OK 2022-11-23T02:51:39.4847664Z 2022-11-23T02:51:39.4847766Z Generating XML reports... 2022-11-23T02:51:39.4848267Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024149.xml 2022-11-23T02:51:39.4848670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4848842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4849249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4849437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4849697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvw_p25z2 2022-11-23T02:51:39.4849966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvw_p25z2/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4850299Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4850305Z 2022-11-23T02:51:39.4850405Z Running tests... 2022-11-23T02:51:39.4850686Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4850948Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4851284Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39150 2022-11-23T02:51:39.4851504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39151 2022-11-23T02:51:39.4851900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4852073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4852485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4852673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4852977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkb7cxtl1 2022-11-23T02:51:39.4853248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkb7cxtl1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4853475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4853867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4854037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4854448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4854636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4854892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpom7q31qm 2022-11-23T02:51:39.4855164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpom7q31qm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4855393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4856270Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:51:39.4856362Z ok (4.031s) 2022-11-23T02:51:39.4856369Z 2022-11-23T02:51:39.4856656Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4856757Z Ran 1 test in 4.031s 2022-11-23T02:51:39.4856764Z 2022-11-23T02:51:39.4856852Z OK 2022-11-23T02:51:39.4856858Z 2022-11-23T02:51:39.4856970Z Generating XML reports... 2022-11-23T02:51:39.4857472Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024156.xml 2022-11-23T02:51:39.4857869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4858041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4858452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4858638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4858897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rxolqno 2022-11-23T02:51:39.4859171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rxolqno/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4859504Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4859512Z 2022-11-23T02:51:39.4859609Z Running tests... 2022-11-23T02:51:39.4859892Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4860251Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39341 2022-11-23T02:51:39.4860468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39342 2022-11-23T02:51:39.4860866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4861037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4861435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4861621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4861927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ioi76ga 2022-11-23T02:51:39.4862198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ioi76ga/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4862432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4862829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4863000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4863404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4863590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4863849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpop2c0x1o 2022-11-23T02:51:39.4864126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpop2c0x1o/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4864352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4864444Z ok (4.119s) 2022-11-23T02:51:39.4864451Z 2022-11-23T02:51:39.4864785Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4864889Z Ran 1 test in 4.120s 2022-11-23T02:51:39.4864897Z 2022-11-23T02:51:39.4864977Z OK 2022-11-23T02:51:39.4864983Z 2022-11-23T02:51:39.4865092Z Generating XML reports... 2022-11-23T02:51:39.4865598Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024203.xml 2022-11-23T02:51:39.4865993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4866164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4866574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4866749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4867005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskpulp8r 2022-11-23T02:51:39.4867282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskpulp8r/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4867612Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4867619Z 2022-11-23T02:51:39.4867715Z Running tests... 2022-11-23T02:51:39.4868000Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4868383Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39532 2022-11-23T02:51:39.4868605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39533 2022-11-23T02:51:39.4868998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4869169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4869577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4869764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4870016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2em2nedx 2022-11-23T02:51:39.4870286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2em2nedx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4870512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4870908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4871125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4871535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4871726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4871984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsuc9bsio 2022-11-23T02:51:39.4872258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsuc9bsio/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4872489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4872570Z ok (4.227s) 2022-11-23T02:51:39.4872589Z 2022-11-23T02:51:39.4872861Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4872967Z Ran 1 test in 4.228s 2022-11-23T02:51:39.4872977Z 2022-11-23T02:51:39.4873056Z OK 2022-11-23T02:51:39.4873063Z 2022-11-23T02:51:39.4873176Z Generating XML reports... 2022-11-23T02:51:39.4873676Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024210.xml 2022-11-23T02:51:39.4874143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4874315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4874728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4874913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4875170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp21ecyay 2022-11-23T02:51:39.4875443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp21ecyay/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4875778Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4875785Z 2022-11-23T02:51:39.4875882Z Running tests... 2022-11-23T02:51:39.4876250Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4876635Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39723 2022-11-23T02:51:39.4876852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39724 2022-11-23T02:51:39.4877251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4877421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4877828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4878014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4878271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vpednbu 2022-11-23T02:51:39.4878533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vpednbu/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4878769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4879163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4879329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4879739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4879928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4880188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3yxy7m8z 2022-11-23T02:51:39.4880523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3yxy7m8z/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4880751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4881774Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4881877Z warnings.warn( 2022-11-23T02:51:39.4882890Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:51:39.4882996Z warnings.warn( 2022-11-23T02:51:39.4883083Z ok (3.947s) 2022-11-23T02:51:39.4883091Z 2022-11-23T02:51:39.4883377Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4883527Z Ran 1 test in 3.948s 2022-11-23T02:51:39.4883535Z 2022-11-23T02:51:39.4883617Z OK 2022-11-23T02:51:39.4883623Z 2022-11-23T02:51:39.4883740Z Generating XML reports... 2022-11-23T02:51:39.4884252Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024217.xml 2022-11-23T02:51:39.4884651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4884822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4885231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4885422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4885683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxohzdl62 2022-11-23T02:51:39.4885960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxohzdl62/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4886291Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4886299Z 2022-11-23T02:51:39.4886385Z Running tests... 2022-11-23T02:51:39.4886669Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4887051Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39914 2022-11-23T02:51:39.4887271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39915 2022-11-23T02:51:39.4887671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4887843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4888255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4888441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4888697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9n39x9j2 2022-11-23T02:51:39.4888973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9n39x9j2/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4889204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4889602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4889818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4890225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4890411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4890669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5d878k_8 2022-11-23T02:51:39.4890941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5d878k_8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4891175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4891414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4891648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4891736Z ok (6.237s) 2022-11-23T02:51:39.4891743Z 2022-11-23T02:51:39.4892030Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4892120Z Ran 1 test in 6.238s 2022-11-23T02:51:39.4892127Z 2022-11-23T02:51:39.4892206Z OK 2022-11-23T02:51:39.4892212Z 2022-11-23T02:51:39.4892324Z Generating XML reports... 2022-11-23T02:51:39.4892873Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024224.xml 2022-11-23T02:51:39.4893281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4893452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4893864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4894051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4894314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnzivnr_ 2022-11-23T02:51:39.4894592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnzivnr_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4894917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4894925Z 2022-11-23T02:51:39.4895020Z Running tests... 2022-11-23T02:51:39.4895303Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4895686Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40107 2022-11-23T02:51:39.4895902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40108 2022-11-23T02:51:39.4896295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4896471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4896887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4897078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4897338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0h94zz0u 2022-11-23T02:51:39.4897610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0h94zz0u/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4897839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4898225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4898395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4898801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4898987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4899289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohwhj1el 2022-11-23T02:51:39.4899562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohwhj1el/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4899794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4900030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4900266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4900360Z ok (6.119s) 2022-11-23T02:51:39.4900367Z 2022-11-23T02:51:39.4900659Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4900764Z Ran 1 test in 6.120s 2022-11-23T02:51:39.4900771Z 2022-11-23T02:51:39.4900854Z OK 2022-11-23T02:51:39.4900860Z 2022-11-23T02:51:39.4900974Z Generating XML reports... 2022-11-23T02:51:39.4901478Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024233.xml 2022-11-23T02:51:39.4901875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4902092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4902503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4902693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4902948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yx26h5r 2022-11-23T02:51:39.4903217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yx26h5r/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4903538Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4903557Z 2022-11-23T02:51:39.4903642Z Running tests... 2022-11-23T02:51:39.4903923Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4904266Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40300 2022-11-23T02:51:39.4904491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40301 2022-11-23T02:51:39.4904885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4905055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4905460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4905649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4905909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_jbv74r 2022-11-23T02:51:39.4906188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_jbv74r/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4906417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4906813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4906990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4907402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4907591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4907849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpou7v8xlz 2022-11-23T02:51:39.4908126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpou7v8xlz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4908402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4908545Z skip: Need at least 4 CUDA devices (3.539s) 2022-11-23T02:51:39.4908553Z 2022-11-23T02:51:39.4908839Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4908944Z Ran 1 test in 3.539s 2022-11-23T02:51:39.4908950Z 2022-11-23T02:51:39.4909035Z OK (skipped=1) 2022-11-23T02:51:39.4909052Z 2022-11-23T02:51:39.4909155Z Generating XML reports... 2022-11-23T02:51:39.4909655Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024242.xml 2022-11-23T02:51:39.4910054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4910220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4910624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4910812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4911066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90hvhglz 2022-11-23T02:51:39.4911385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90hvhglz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4911719Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4911726Z 2022-11-23T02:51:39.4911823Z Running tests... 2022-11-23T02:51:39.4912107Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4912450Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40477 2022-11-23T02:51:39.4912672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40478 2022-11-23T02:51:39.4913073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4913240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4913648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4913836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4914095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunch3ha6 2022-11-23T02:51:39.4914368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunch3ha6/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4914598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4914996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4915155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4915564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4915748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4916009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdhtc00w 2022-11-23T02:51:39.4916392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdhtc00w/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4916624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4916767Z skip: Need at least 8 CUDA devices (3.718s) 2022-11-23T02:51:39.4916774Z 2022-11-23T02:51:39.4917060Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4917161Z Ran 1 test in 3.719s 2022-11-23T02:51:39.4917170Z 2022-11-23T02:51:39.4917266Z OK (skipped=1) 2022-11-23T02:51:39.4917337Z 2022-11-23T02:51:39.4917455Z Generating XML reports... 2022-11-23T02:51:39.4917964Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024249.xml 2022-11-23T02:51:39.4918359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4918533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4918943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4919135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4919397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfy5sh0m 2022-11-23T02:51:39.4919672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfy5sh0m/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4920002Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4920013Z 2022-11-23T02:51:39.4920114Z Running tests... 2022-11-23T02:51:39.4920400Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4920795Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40654 2022-11-23T02:51:39.4921008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40655 2022-11-23T02:51:39.4921406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4921575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4921987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4922173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4922433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdl80911x 2022-11-23T02:51:39.4922704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdl80911x/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4923105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4923277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4923682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4923870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4924127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtmznh_1 2022-11-23T02:51:39.4924401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtmznh_1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4924631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4924863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4925099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4925337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4925423Z ok (3.418s) 2022-11-23T02:51:39.4925430Z 2022-11-23T02:51:39.4925711Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4925810Z Ran 1 test in 3.419s 2022-11-23T02:51:39.4925816Z 2022-11-23T02:51:39.4925896Z OK 2022-11-23T02:51:39.4925902Z 2022-11-23T02:51:39.4926005Z Generating XML reports... 2022-11-23T02:51:39.4926510Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024256.xml 2022-11-23T02:51:39.4926906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4927130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4927547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4927740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4927995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_naa9bl8 2022-11-23T02:51:39.4928270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_naa9bl8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4928603Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4928610Z 2022-11-23T02:51:39.4928705Z Running tests... 2022-11-23T02:51:39.4928988Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4929349Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40841 2022-11-23T02:51:39.4929571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40842 2022-11-23T02:51:39.4930009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4930180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4930589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4930775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4931034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodumv36y 2022-11-23T02:51:39.4931309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodumv36y/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4931539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4931937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4932104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4932506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4932695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4932953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qxq8wug 2022-11-23T02:51:39.4933222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qxq8wug/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4933450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4933687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4933923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4934013Z ok (3.723s) 2022-11-23T02:51:39.4934021Z 2022-11-23T02:51:39.4934310Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4934410Z Ran 1 test in 3.724s 2022-11-23T02:51:39.4934420Z 2022-11-23T02:51:39.4934500Z OK 2022-11-23T02:51:39.4934507Z 2022-11-23T02:51:39.4934621Z Generating XML reports... 2022-11-23T02:51:39.4935123Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024302.xml 2022-11-23T02:51:39.4935520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4935692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4936097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4936331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4936585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jl0qvsu 2022-11-23T02:51:39.4936859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jl0qvsu/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4937199Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4937206Z 2022-11-23T02:51:39.4937305Z Running tests... 2022-11-23T02:51:39.4937579Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4937775Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4938096Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41028 2022-11-23T02:51:39.4938311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41029 2022-11-23T02:51:39.4938711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4938880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4939333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4939520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4939778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsf5zxg2s 2022-11-23T02:51:39.4940055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsf5zxg2s/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4940284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4940680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4940856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4941262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4941453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4941710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qkj83j3 2022-11-23T02:51:39.4941985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qkj83j3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4942215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4942451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4942686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4942773Z ok (3.933s) 2022-11-23T02:51:39.4942783Z 2022-11-23T02:51:39.4943065Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4943155Z Ran 1 test in 3.934s 2022-11-23T02:51:39.4943161Z 2022-11-23T02:51:39.4943241Z OK 2022-11-23T02:51:39.4943247Z 2022-11-23T02:51:39.4943356Z Generating XML reports... 2022-11-23T02:51:39.4943856Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024309.xml 2022-11-23T02:51:39.4944250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4944418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4944819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4945002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4945261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqv3si75 2022-11-23T02:51:39.4945606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqv3si75/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4945941Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4945948Z 2022-11-23T02:51:39.4946043Z Running tests... 2022-11-23T02:51:39.4946328Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4946557Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-11-23T02:51:39.4946872Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41269 2022-11-23T02:51:39.4947091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41270 2022-11-23T02:51:39.4947491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4947668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4948080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4948270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4948572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkm8uod80 2022-11-23T02:51:39.4948837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkm8uod80/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4949070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4949473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4949644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4950050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4950240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4950495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9363va3 2022-11-23T02:51:39.4950770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9363va3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4950998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4951087Z ok (3.657s) 2022-11-23T02:51:39.4951094Z 2022-11-23T02:51:39.4951381Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4951486Z Ran 1 test in 3.658s 2022-11-23T02:51:39.4951494Z 2022-11-23T02:51:39.4951575Z OK 2022-11-23T02:51:39.4951581Z 2022-11-23T02:51:39.4951700Z Generating XML reports... 2022-11-23T02:51:39.4952202Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024316.xml 2022-11-23T02:51:39.4952606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4952780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4953189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4953374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4953628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9s1v4dtz 2022-11-23T02:51:39.4953898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9s1v4dtz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4954229Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4954237Z 2022-11-23T02:51:39.4954322Z Running tests... 2022-11-23T02:51:39.4954609Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4954998Z test_ignored_sharded_tensor (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41510 2022-11-23T02:51:39.4955214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41511 2022-11-23T02:51:39.4955618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4955787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4956264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4956454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4956714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_2fs5n2 2022-11-23T02:51:39.4956985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_2fs5n2/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4957211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4957606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4957832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4958242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4958431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4958691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbug9vcaa 2022-11-23T02:51:39.4958970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbug9vcaa/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4959201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4959454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4959697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4960135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4960568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4960650Z ok (3.716s) 2022-11-23T02:51:39.4960670Z 2022-11-23T02:51:39.4960948Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4961051Z Ran 1 test in 3.725s 2022-11-23T02:51:39.4961058Z 2022-11-23T02:51:39.4961141Z OK 2022-11-23T02:51:39.4961148Z 2022-11-23T02:51:39.4961264Z Generating XML reports... 2022-11-23T02:51:39.4961770Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024322.xml 2022-11-23T02:51:39.4962170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4962339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4962749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4962938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4963197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5wa25_c 2022-11-23T02:51:39.4963474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5wa25_c/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4963807Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4963815Z 2022-11-23T02:51:39.4963909Z Running tests... 2022-11-23T02:51:39.4964253Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4964595Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41693 2022-11-23T02:51:39.4964821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41694 2022-11-23T02:51:39.4965229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4965401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4965811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4965999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4966257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkjzv6yh 2022-11-23T02:51:39.4966523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkjzv6yh/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4966753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4967377Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4967970Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4968535Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4969117Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4969679Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4970262Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4970671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4970845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4971263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4971457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4971715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5koesxnx 2022-11-23T02:51:39.4971987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5koesxnx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4972265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4972840Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4973406Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4973975Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4974677Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4975247Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4975816Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:51:39.4975910Z ok (3.525s) 2022-11-23T02:51:39.4975918Z 2022-11-23T02:51:39.4976213Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4976317Z Ran 1 test in 3.526s 2022-11-23T02:51:39.4976325Z 2022-11-23T02:51:39.4976411Z OK 2022-11-23T02:51:39.4976418Z 2022-11-23T02:51:39.4976534Z Generating XML reports... 2022-11-23T02:51:39.4977047Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024329.xml 2022-11-23T02:51:39.4977446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4977628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4978047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4978241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4978494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83bysjvg 2022-11-23T02:51:39.4978771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83bysjvg/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4979108Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4979115Z 2022-11-23T02:51:39.4979209Z Running tests... 2022-11-23T02:51:39.4979493Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4979838Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41870 2022-11-23T02:51:39.4980106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41871 2022-11-23T02:51:39.4980511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4980685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4981097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4981285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4981542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl96osoes 2022-11-23T02:51:39.4981816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl96osoes/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4982047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4982447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4982617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4983072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4983269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4983527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_y3n50kb 2022-11-23T02:51:39.4983803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_y3n50kb/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4984037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4984288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.4984721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4984963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.4985395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:39.4985637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4985863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4986097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4986323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.4986411Z ok (6.276s) 2022-11-23T02:51:39.4986418Z 2022-11-23T02:51:39.4986705Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4986810Z Ran 1 test in 6.277s 2022-11-23T02:51:39.4986816Z 2022-11-23T02:51:39.4986897Z OK 2022-11-23T02:51:39.4986903Z 2022-11-23T02:51:39.4987015Z Generating XML reports... 2022-11-23T02:51:39.4987527Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024336.xml 2022-11-23T02:51:39.4987926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4988102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4988515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4988704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4988964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsadvevo6 2022-11-23T02:51:39.4989237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsadvevo6/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4989627Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4989634Z 2022-11-23T02:51:39.4989732Z Running tests... 2022-11-23T02:51:39.4990020Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4990345Z test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42063 2022-11-23T02:51:39.4990567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42064 2022-11-23T02:51:39.4990961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4991135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4991547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4991739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4991999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mhq7e1c 2022-11-23T02:51:39.4992323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mhq7e1c/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4992554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.4992955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4993133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4993544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4993733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4993995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ug828e_ 2022-11-23T02:51:39.4994262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ug828e_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4994491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.4994584Z ok (3.628s) 2022-11-23T02:51:39.4994591Z 2022-11-23T02:51:39.4994875Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4994979Z Ran 1 test in 3.630s 2022-11-23T02:51:39.4994986Z 2022-11-23T02:51:39.4995064Z OK 2022-11-23T02:51:39.4995070Z 2022-11-23T02:51:39.4995184Z Generating XML reports... 2022-11-23T02:51:39.4995676Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024345.xml 2022-11-23T02:51:39.4996138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4996314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4996726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4996913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.4997169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40wae838 2022-11-23T02:51:39.4997436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40wae838/_remote_module_non_scriptable.py 2022-11-23T02:51:39.4997767Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.4997775Z 2022-11-23T02:51:39.4997869Z Running tests... 2022-11-23T02:51:39.4998154Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.4998502Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42304 2022-11-23T02:51:39.4998783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42305 2022-11-23T02:51:39.4999182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.4999356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.4999767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.4999953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5000214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdc7nque 2022-11-23T02:51:39.5000492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdc7nque/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5000724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5001125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5001298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5001708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5001942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5002188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8toit5l 2022-11-23T02:51:39.5002455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8toit5l/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5002684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5002777Z ok (3.698s) 2022-11-23T02:51:39.5002784Z 2022-11-23T02:51:39.5003072Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5003177Z Ran 1 test in 3.699s 2022-11-23T02:51:39.5003185Z 2022-11-23T02:51:39.5003263Z OK 2022-11-23T02:51:39.5003269Z 2022-11-23T02:51:39.5003380Z Generating XML reports... 2022-11-23T02:51:39.5003886Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024352.xml 2022-11-23T02:51:39.5004289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5004461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5004868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5005055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5005310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47utdsa6 2022-11-23T02:51:39.5005582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47utdsa6/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5005916Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5005923Z 2022-11-23T02:51:39.5006021Z Running tests... 2022-11-23T02:51:39.5006305Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5006651Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42545 2022-11-23T02:51:39.5006871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42546 2022-11-23T02:51:39.5007269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5007432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5007845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5008081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5008340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbqsbs56 2022-11-23T02:51:39.5008615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbqsbs56/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5008846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5009238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5009409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5009814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5009998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5010251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwks002m 2022-11-23T02:51:39.5010527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwks002m/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5010756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5011061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.5011297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.5011389Z ok (7.455s) 2022-11-23T02:51:39.5011396Z 2022-11-23T02:51:39.5011682Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5011783Z Ran 1 test in 7.455s 2022-11-23T02:51:39.5011790Z 2022-11-23T02:51:39.5011868Z OK 2022-11-23T02:51:39.5011874Z 2022-11-23T02:51:39.5011988Z Generating XML reports... 2022-11-23T02:51:39.5012495Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024359.xml 2022-11-23T02:51:39.5012885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5013056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5013468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5013656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5013914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppj24zxx9 2022-11-23T02:51:39.5014184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppj24zxx9/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5014512Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5014519Z 2022-11-23T02:51:39.5014618Z Running tests... 2022-11-23T02:51:39.5014906Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5015259Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42777 2022-11-23T02:51:39.5015479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42778 2022-11-23T02:51:39.5015880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5016057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5016468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5016655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5016908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprv7hl3a8 2022-11-23T02:51:39.5017181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprv7hl3a8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5017500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5017901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5018075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5018483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5018675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5018923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6glo_kvt 2022-11-23T02:51:39.5019194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6glo_kvt/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5019422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5019661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.5019893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:51:39.5019983Z ok (4.915s) 2022-11-23T02:51:39.5019990Z 2022-11-23T02:51:39.5020319Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5020421Z Ran 1 test in 4.916s 2022-11-23T02:51:39.5020428Z 2022-11-23T02:51:39.5020507Z OK 2022-11-23T02:51:39.5020514Z 2022-11-23T02:51:39.5020625Z Generating XML reports... 2022-11-23T02:51:39.5021128Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123024409.xml 2022-11-23T02:51:39.5021527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5021695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5022108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5022293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5022550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ngaxlrc 2022-11-23T02:51:39.5022826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ngaxlrc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5023156Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5023164Z 2022-11-23T02:51:39.5023258Z Running tests... 2022-11-23T02:51:39.5023542Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5023955Z test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42968 2022-11-23T02:51:39.5024355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5024515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5024925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5025113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5025373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl56dv1ii 2022-11-23T02:51:39.5025646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl56dv1ii/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5025875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5026121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.5026562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:51:39.5027455Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5027562Z warnings.warn( 2022-11-23T02:51:39.5027650Z ok (3.568s) 2022-11-23T02:51:39.5027657Z 2022-11-23T02:51:39.5027942Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5028041Z Ran 1 test in 3.568s 2022-11-23T02:51:39.5028048Z 2022-11-23T02:51:39.5028127Z OK 2022-11-23T02:51:39.5028134Z 2022-11-23T02:51:39.5028246Z Generating XML reports... 2022-11-23T02:51:39.5028853Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123024417.xml 2022-11-23T02:51:39.5029256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5029426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5029830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5030059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5030319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9b7oilb3 2022-11-23T02:51:39.5030593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9b7oilb3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5030928Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5030935Z 2022-11-23T02:51:39.5031034Z Running tests... 2022-11-23T02:51:39.5031310Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5031728Z test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43089 2022-11-23T02:51:39.5032127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5032304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5032714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5032904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5033162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexx5dvr6 2022-11-23T02:51:39.5033433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexx5dvr6/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5033659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5033912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.5034345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:51:39.5035169Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5035271Z warnings.warn( 2022-11-23T02:51:39.5035358Z ok (3.319s) 2022-11-23T02:51:39.5035365Z 2022-11-23T02:51:39.5035649Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5035748Z Ran 1 test in 3.320s 2022-11-23T02:51:39.5035755Z 2022-11-23T02:51:39.5035833Z OK 2022-11-23T02:51:39.5035839Z 2022-11-23T02:51:39.5035949Z Generating XML reports... 2022-11-23T02:51:39.5036676Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123024424.xml 2022-11-23T02:51:39.5037079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5037254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5037661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5037847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5038094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv64bmcc 2022-11-23T02:51:39.5038364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv64bmcc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5038698Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5038707Z 2022-11-23T02:51:39.5038804Z Running tests... 2022-11-23T02:51:39.5039091Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5039542Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43210 2022-11-23T02:51:39.5039951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5040129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5040543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5040737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5041000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8nimyqa 2022-11-23T02:51:39.5048040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8nimyqa/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5048314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5048568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.5049054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:51:39.5049142Z ok (3.364s) 2022-11-23T02:51:39.5049151Z 2022-11-23T02:51:39.5049438Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5049539Z Ran 1 test in 3.365s 2022-11-23T02:51:39.5049547Z 2022-11-23T02:51:39.5049623Z OK 2022-11-23T02:51:39.5049629Z 2022-11-23T02:51:39.5049740Z Generating XML reports... 2022-11-23T02:51:39.5050357Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123024430.xml 2022-11-23T02:51:39.5050767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5050931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5051344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5051533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5051790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2n8zqyq 2022-11-23T02:51:39.5052061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2n8zqyq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5052394Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5052401Z 2022-11-23T02:51:39.5052501Z Running tests... 2022-11-23T02:51:39.5052787Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5053329Z test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43331 2022-11-23T02:51:39.5053733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5053904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5054313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5054498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5054752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw_f3k2_ 2022-11-23T02:51:39.5055021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw_f3k2_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5055244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5055496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.5055930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:51:39.5056074Z ok (3.311s) 2022-11-23T02:51:39.5056082Z 2022-11-23T02:51:39.5056368Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5056465Z Ran 1 test in 3.312s 2022-11-23T02:51:39.5056473Z 2022-11-23T02:51:39.5056548Z OK 2022-11-23T02:51:39.5056554Z 2022-11-23T02:51:39.5056662Z Generating XML reports... 2022-11-23T02:51:39.5057258Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123024436.xml 2022-11-23T02:51:39.5057654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5057826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5058235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5058421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5058677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc84ae5o1 2022-11-23T02:51:39.5058950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc84ae5o1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5059278Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5059286Z 2022-11-23T02:51:39.5059378Z Running tests... 2022-11-23T02:51:39.5059657Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5059971Z test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43452 2022-11-23T02:51:39.5060189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43453 2022-11-23T02:51:39.5060402Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43454 2022-11-23T02:51:39.5060612Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43455 2022-11-23T02:51:39.5061009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5061180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5061584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5061769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5062023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07jctwbr 2022-11-23T02:51:39.5062418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5062631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5063027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5063214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5063466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9tfvrmw9 2022-11-23T02:51:39.5063736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07jctwbr/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5063998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9tfvrmw9/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5064225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5064445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5064848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5065014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5065470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5065656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5065912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr979zfj7 2022-11-23T02:51:39.5066179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr979zfj7/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5066405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5066799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5066966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5067373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5067555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5067809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxhf66so 2022-11-23T02:51:39.5068080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxhf66so/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5068302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5068388Z ok (4.329s) 2022-11-23T02:51:39.5068395Z 2022-11-23T02:51:39.5068669Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5068766Z Ran 1 test in 4.329s 2022-11-23T02:51:39.5068773Z 2022-11-23T02:51:39.5068850Z OK 2022-11-23T02:51:39.5068859Z 2022-11-23T02:51:39.5068969Z Generating XML reports... 2022-11-23T02:51:39.5069439Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024443.xml 2022-11-23T02:51:39.5069833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5070004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5070406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5070589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5070843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgy2xkuzd 2022-11-23T02:51:39.5071117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgy2xkuzd/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5071443Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5071495Z 2022-11-23T02:51:39.5071592Z Running tests... 2022-11-23T02:51:39.5071876Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5072198Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43759 2022-11-23T02:51:39.5072414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43760 2022-11-23T02:51:39.5072625Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43761 2022-11-23T02:51:39.5072833Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43762 2022-11-23T02:51:39.5073226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5073392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5073795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5073974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5074228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyy32gugm 2022-11-23T02:51:39.5074544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyy32gugm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5074771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5075167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5075334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5075739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5075923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5076256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51en2e56 2022-11-23T02:51:39.5076527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51en2e56/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5076927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5077097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5077498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5077682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5077934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbep2tit3 2022-11-23T02:51:39.5078206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbep2tit3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5078435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5078654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5079054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5079221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5079625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5079809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5080053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht5t4e04 2022-11-23T02:51:39.5080317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht5t4e04/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5080551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5080747Z ok (4.836s) 2022-11-23T02:51:39.5080755Z 2022-11-23T02:51:39.5081037Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5081135Z Ran 1 test in 4.839s 2022-11-23T02:51:39.5081142Z 2022-11-23T02:51:39.5081218Z OK 2022-11-23T02:51:39.5081228Z 2022-11-23T02:51:39.5081338Z Generating XML reports... 2022-11-23T02:51:39.5081804Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024450.xml 2022-11-23T02:51:39.5082197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5082364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5082769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5082953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5083213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7io7ueph 2022-11-23T02:51:39.5083484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7io7ueph/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5083862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5083870Z 2022-11-23T02:51:39.5083964Z Running tests... 2022-11-23T02:51:39.5084250Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5084564Z test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44066 2022-11-23T02:51:39.5084776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44067 2022-11-23T02:51:39.5084987Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44068 2022-11-23T02:51:39.5085195Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44069 2022-11-23T02:51:39.5085586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5085753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5086158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5086345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5086595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nx634qm 2022-11-23T02:51:39.5086867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nx634qm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5087093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5087484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5087653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5088062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5088247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5088502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgop1ogw0 2022-11-23T02:51:39.5088770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgop1ogw0/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5089159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5089325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5089729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5089982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5090233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49wbth_c 2022-11-23T02:51:39.5090506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49wbth_c/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5090734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5090959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5091348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5091516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5091918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5092102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5092353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0232om2h 2022-11-23T02:51:39.5092616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0232om2h/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5092883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5092974Z ok (4.475s) 2022-11-23T02:51:39.5092981Z 2022-11-23T02:51:39.5093265Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5093362Z Ran 1 test in 4.476s 2022-11-23T02:51:39.5093369Z 2022-11-23T02:51:39.5093448Z OK 2022-11-23T02:51:39.5093454Z 2022-11-23T02:51:39.5093569Z Generating XML reports... 2022-11-23T02:51:39.5094043Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024458.xml 2022-11-23T02:51:39.5094439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5094613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5095017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5095203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5095455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcxs9ioj 2022-11-23T02:51:39.5095722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcxs9ioj/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5096050Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5096057Z 2022-11-23T02:51:39.5096151Z Running tests... 2022-11-23T02:51:39.5096424Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5096752Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44373 2022-11-23T02:51:39.5096970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44374 2022-11-23T02:51:39.5097182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44375 2022-11-23T02:51:39.5097392Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44376 2022-11-23T02:51:39.5097785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5097950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5098353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5098536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5098785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxa2zpiv 2022-11-23T02:51:39.5099110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxa2zpiv/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5099333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5099729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5099894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5100296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5100480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5100734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4n1pdfc 2022-11-23T02:51:39.5101007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4n1pdfc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5101406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5101575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5102020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5102196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5102448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzk6b1a9k 2022-11-23T02:51:39.5102713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzk6b1a9k/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5102934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5103149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5103548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5103722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5104126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5104311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5104564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6u6vu8jp 2022-11-23T02:51:39.5104835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6u6vu8jp/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5105059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5105304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.5105546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.5105789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:39.5106025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:39.5106462Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:39.5106891Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:39.5107317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:39.5107739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:39.5108569Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5108712Z warnings.warn( 2022-11-23T02:51:39.5109530Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5109626Z warnings.warn( 2022-11-23T02:51:39.5110426Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5110520Z warnings.warn( 2022-11-23T02:51:39.5111315Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5111412Z warnings.warn( 2022-11-23T02:51:39.5111489Z ok (4.439s) 2022-11-23T02:51:39.5111545Z 2022-11-23T02:51:39.5111825Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5111923Z Ran 1 test in 4.440s 2022-11-23T02:51:39.5111930Z 2022-11-23T02:51:39.5112006Z OK 2022-11-23T02:51:39.5112013Z 2022-11-23T02:51:39.5112122Z Generating XML reports... 2022-11-23T02:51:39.5112591Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024506.xml 2022-11-23T02:51:39.5112986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5113152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5113557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5113742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5113998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptio_f15u 2022-11-23T02:51:39.5114266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptio_f15u/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5114598Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5114606Z 2022-11-23T02:51:39.5114699Z Running tests... 2022-11-23T02:51:39.5114980Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5115311Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44680 2022-11-23T02:51:39.5115533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44681 2022-11-23T02:51:39.5115744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44682 2022-11-23T02:51:39.5115951Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44683 2022-11-23T02:51:39.5116420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5116590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5116998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5117175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5117428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzkeluwz 2022-11-23T02:51:39.5117699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzkeluwz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5118154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5118321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5118723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5118905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5119154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4n2y88e 2022-11-23T02:51:39.5119422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4n2y88e/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5119648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5119871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5120267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5120431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5120881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5121066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5121321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjxr0f_s 2022-11-23T02:51:39.5121594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjxr0f_s/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5121981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5122147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5122553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5122738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5122990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpux8a_dsu 2022-11-23T02:51:39.5123253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpux8a_dsu/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5123482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5123709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5124522Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5124626Z warnings.warn( 2022-11-23T02:51:39.5125434Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5125532Z warnings.warn( 2022-11-23T02:51:39.5126337Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5126432Z warnings.warn( 2022-11-23T02:51:39.5127223Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5127371Z warnings.warn( 2022-11-23T02:51:39.5127455Z ok (5.033s) 2022-11-23T02:51:39.5127462Z 2022-11-23T02:51:39.5127746Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5127843Z Ran 1 test in 5.033s 2022-11-23T02:51:39.5127850Z 2022-11-23T02:51:39.5127926Z OK 2022-11-23T02:51:39.5127936Z 2022-11-23T02:51:39.5128047Z Generating XML reports... 2022-11-23T02:51:39.5128512Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024513.xml 2022-11-23T02:51:39.5128905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5129072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5129473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5129660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5129908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_nddcz1s 2022-11-23T02:51:39.5130177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_nddcz1s/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5130548Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5130556Z 2022-11-23T02:51:39.5130649Z Running tests... 2022-11-23T02:51:39.5130927Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5131254Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44987 2022-11-23T02:51:39.5131469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44988 2022-11-23T02:51:39.5131683Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44989 2022-11-23T02:51:39.5131896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44990 2022-11-23T02:51:39.5132291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5132457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5132867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5133050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5133304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6q9t_9a 2022-11-23T02:51:39.5133570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6q9t_9a/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5133800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5134189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5134358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5134760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5134944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5135196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4ndf9te 2022-11-23T02:51:39.5135461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4ndf9te/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5135853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5136019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5136417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5136640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5136884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqt59izj 2022-11-23T02:51:39.5137154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqt59izj/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5137381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5137607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5137998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5138166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5138567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5138753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5139005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmput7vnvf8 2022-11-23T02:51:39.5139272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmput7vnvf8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5139537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5139626Z ok (4.529s) 2022-11-23T02:51:39.5139633Z 2022-11-23T02:51:39.5139916Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5140013Z Ran 1 test in 4.530s 2022-11-23T02:51:39.5140020Z 2022-11-23T02:51:39.5140094Z OK 2022-11-23T02:51:39.5140101Z 2022-11-23T02:51:39.5140211Z Generating XML reports... 2022-11-23T02:51:39.5140677Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024521.xml 2022-11-23T02:51:39.5141071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5141240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5141645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5141830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5142080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp235a59l1 2022-11-23T02:51:39.5142338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp235a59l1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5142664Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5142671Z 2022-11-23T02:51:39.5142764Z Running tests... 2022-11-23T02:51:39.5143043Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5143356Z test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45294 2022-11-23T02:51:39.5143572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45295 2022-11-23T02:51:39.5143780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45296 2022-11-23T02:51:39.5143992Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45297 2022-11-23T02:51:39.5144383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5144551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5144949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5145133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5145387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplb9b00ts 2022-11-23T02:51:39.5145703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplb9b00ts/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5145928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5146326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5146493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5146898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5147081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5147335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbh333hpb 2022-11-23T02:51:39.5147607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbh333hpb/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5147829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5148222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5148387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5148836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5149023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5149274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrbboqhg 2022-11-23T02:51:39.5149543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrbboqhg/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5149766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5150162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5150329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5150729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5150918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5151170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaopy3jyv 2022-11-23T02:51:39.5151440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaopy3jyv/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5151663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5151749Z ok (4.927s) 2022-11-23T02:51:39.5151756Z 2022-11-23T02:51:39.5152037Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5152137Z Ran 1 test in 4.927s 2022-11-23T02:51:39.5152144Z 2022-11-23T02:51:39.5152219Z OK 2022-11-23T02:51:39.5152226Z 2022-11-23T02:51:39.5152335Z Generating XML reports... 2022-11-23T02:51:39.5152797Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024529.xml 2022-11-23T02:51:39.5153184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5153352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5153760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5153945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5154205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuuyv7dr 2022-11-23T02:51:39.5154472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuuyv7dr/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5154845Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5154852Z 2022-11-23T02:51:39.5154946Z Running tests... 2022-11-23T02:51:39.5155226Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5155549Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45625 2022-11-23T02:51:39.5155768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45626 2022-11-23T02:51:39.5155979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45627 2022-11-23T02:51:39.5156237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45628 2022-11-23T02:51:39.5156632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5156800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5157206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5157389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5157712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvou4a0r3 2022-11-23T02:51:39.5157982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvou4a0r3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5158210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5158603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5158769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5159165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5159352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5159603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21_bkf4w 2022-11-23T02:51:39.5159872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21_bkf4w/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5160096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5160489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5160655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5161061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5161243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5161494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6bg965j 2022-11-23T02:51:39.5161764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6bg965j/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5161995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5162389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5162557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5162960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5163140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5163396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3u9pnel 2022-11-23T02:51:39.5163666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3u9pnel/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5163967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5164052Z ok (7.308s) 2022-11-23T02:51:39.5164059Z 2022-11-23T02:51:39.5164345Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5164438Z Ran 1 test in 7.309s 2022-11-23T02:51:39.5164452Z 2022-11-23T02:51:39.5164521Z OK 2022-11-23T02:51:39.5164527Z 2022-11-23T02:51:39.5164636Z Generating XML reports... 2022-11-23T02:51:39.5165102Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024537.xml 2022-11-23T02:51:39.5165495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5165663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5166065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5166251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5166502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcv6ma7ld 2022-11-23T02:51:39.5166813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcv6ma7ld/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5167144Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5167152Z 2022-11-23T02:51:39.5167242Z Running tests... 2022-11-23T02:51:39.5167520Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5167839Z test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45956 2022-11-23T02:51:39.5168051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45957 2022-11-23T02:51:39.5168260Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45958 2022-11-23T02:51:39.5168469Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45959 2022-11-23T02:51:39.5168856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5169027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5169428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5169609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5169859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylzzvmas 2022-11-23T02:51:39.5170126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylzzvmas/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5170510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5170678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5171081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5171262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5171512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuee_1nsq 2022-11-23T02:51:39.5171776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuee_1nsq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5171999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5172391Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5172556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5172955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5173182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5173432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe1vc_en 2022-11-23T02:51:39.5173704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe1vc_en/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5173928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5174150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5174540Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5174706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5175106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5175292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5175539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_smco7f 2022-11-23T02:51:39.5175840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_smco7f/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5176063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5176149Z ok (4.827s) 2022-11-23T02:51:39.5176156Z 2022-11-23T02:51:39.5176440Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5176540Z Ran 1 test in 4.827s 2022-11-23T02:51:39.5176547Z 2022-11-23T02:51:39.5176623Z OK 2022-11-23T02:51:39.5176630Z 2022-11-23T02:51:39.5176740Z Generating XML reports... 2022-11-23T02:51:39.5177204Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024547.xml 2022-11-23T02:51:39.5177602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5177767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5178177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5178361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5178615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjfqd8tn 2022-11-23T02:51:39.5178888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjfqd8tn/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5179217Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5179224Z 2022-11-23T02:51:39.5179314Z Running tests... 2022-11-23T02:51:39.5179591Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5179919Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46263 2022-11-23T02:51:39.5180133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46264 2022-11-23T02:51:39.5180348Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46265 2022-11-23T02:51:39.5180556Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46266 2022-11-23T02:51:39.5180950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5181111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5181515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5181695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5181997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf1hscg1 2022-11-23T02:51:39.5182269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf1hscg1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5182493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5182889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5183055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5183460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5183643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5183895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5xqzmlo 2022-11-23T02:51:39.5184164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5xqzmlo/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5184391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5184780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5185071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5185478Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5185660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5185914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuiezisb 2022-11-23T02:51:39.5186183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuiezisb/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5186576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5186751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5187147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5187333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5187589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyumlpsm 2022-11-23T02:51:39.5187857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyumlpsm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5188083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5188304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5188393Z ok (5.191s) 2022-11-23T02:51:39.5188400Z 2022-11-23T02:51:39.5188684Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5188785Z Ran 1 test in 5.192s 2022-11-23T02:51:39.5188791Z 2022-11-23T02:51:39.5188867Z OK 2022-11-23T02:51:39.5188873Z 2022-11-23T02:51:39.5188982Z Generating XML reports... 2022-11-23T02:51:39.5189453Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024555.xml 2022-11-23T02:51:39.5189845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5190013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5190416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5190598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5190853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeb7qw36r 2022-11-23T02:51:39.5191165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeb7qw36r/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5191495Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5191502Z 2022-11-23T02:51:39.5191591Z Running tests... 2022-11-23T02:51:39.5191866Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5192198Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46570 2022-11-23T02:51:39.5192413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46571 2022-11-23T02:51:39.5192623Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46572 2022-11-23T02:51:39.5192829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46573 2022-11-23T02:51:39.5193223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5193392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5193794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5194019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5194276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazeyclhp 2022-11-23T02:51:39.5194547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazeyclhp/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5194771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5195165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5195331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5195738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5195920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5196235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2490l9_1 2022-11-23T02:51:39.5196506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2490l9_1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5196900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5197067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5197471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5197651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5197904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbrru22q 2022-11-23T02:51:39.5198167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbrru22q/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5198390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5198618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5199012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5199180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5199583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5199764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5200016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpah2ntky5 2022-11-23T02:51:39.5200337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpah2ntky5/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5200559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5200647Z ok (5.431s) 2022-11-23T02:51:39.5200655Z 2022-11-23T02:51:39.5200943Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5201045Z Ran 1 test in 5.432s 2022-11-23T02:51:39.5201052Z 2022-11-23T02:51:39.5201128Z OK 2022-11-23T02:51:39.5201134Z 2022-11-23T02:51:39.5201242Z Generating XML reports... 2022-11-23T02:51:39.5201713Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024603.xml 2022-11-23T02:51:39.5202107Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5202271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5202677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5202860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5203112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzazurzo 2022-11-23T02:51:39.5203426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzazurzo/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5203756Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5203770Z 2022-11-23T02:51:39.5203856Z Running tests... 2022-11-23T02:51:39.5204139Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5204471Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46877 2022-11-23T02:51:39.5204689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46878 2022-11-23T02:51:39.5204905Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46879 2022-11-23T02:51:39.5205114Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46880 2022-11-23T02:51:39.5205510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5205677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5206079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5206266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5206517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i_gnt8e 2022-11-23T02:51:39.5206785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i_gnt8e/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5207179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5207347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5207751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5207933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5208192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7vdg274 2022-11-23T02:51:39.5208458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7vdg274/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5208844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5209011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5209410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5209638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5209890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpualub36y 2022-11-23T02:51:39.5210162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpualub36y/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5210387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5210610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5210825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5211217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5211382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5211785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5211967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5212267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdo1b6u4 2022-11-23T02:51:39.5212540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdo1b6u4/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5212762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5212850Z ok (4.343s) 2022-11-23T02:51:39.5212858Z 2022-11-23T02:51:39.5213142Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5213241Z Ran 1 test in 4.344s 2022-11-23T02:51:39.5213249Z 2022-11-23T02:51:39.5213326Z OK 2022-11-23T02:51:39.5213332Z 2022-11-23T02:51:39.5213441Z Generating XML reports... 2022-11-23T02:51:39.5213909Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024612.xml 2022-11-23T02:51:39.5214301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5214461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5214865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5215050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5215304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hkjhnrp 2022-11-23T02:51:39.5215577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hkjhnrp/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5215904Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5215911Z 2022-11-23T02:51:39.5216010Z Running tests... 2022-11-23T02:51:39.5216287Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5216605Z test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47184 2022-11-23T02:51:39.5216821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47185 2022-11-23T02:51:39.5217037Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47186 2022-11-23T02:51:39.5217244Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47187 2022-11-23T02:51:39.5217638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5217808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5218213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5218445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5218702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyytfukz 2022-11-23T02:51:39.5218972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyytfukz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5219204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5219595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5219765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5220172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5220348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5220600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn55q45y7 2022-11-23T02:51:39.5220868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn55q45y7/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5221090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5221525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5221693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5222102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5222284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5222537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomtzcuuf 2022-11-23T02:51:39.5222805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomtzcuuf/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5223033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5223426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5223597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5224001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5224183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5224435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifj47a3_ 2022-11-23T02:51:39.5224700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifj47a3_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5224925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5225014Z ok (4.457s) 2022-11-23T02:51:39.5225022Z 2022-11-23T02:51:39.5225305Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5225403Z Ran 1 test in 4.458s 2022-11-23T02:51:39.5225410Z 2022-11-23T02:51:39.5225478Z OK 2022-11-23T02:51:39.5225491Z 2022-11-23T02:51:39.5225593Z Generating XML reports... 2022-11-23T02:51:39.5226061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024619.xml 2022-11-23T02:51:39.5226455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5226622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5227030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5227212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5227469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzemam1e 2022-11-23T02:51:39.5227810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzemam1e/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5228142Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5228150Z 2022-11-23T02:51:39.5228246Z Running tests... 2022-11-23T02:51:39.5228523Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5228855Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47491 2022-11-23T02:51:39.5229070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47492 2022-11-23T02:51:39.5229287Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47493 2022-11-23T02:51:39.5229497Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47494 2022-11-23T02:51:39.5229898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5230065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5230514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5230699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5230952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxiyan5f0 2022-11-23T02:51:39.5231215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxiyan5f0/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5231438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5231836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5232005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5232412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5232596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5232852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkokie9n0 2022-11-23T02:51:39.5233122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkokie9n0/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5233513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5233680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5234078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5234261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5234517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0uys770y 2022-11-23T02:51:39.5234783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0uys770y/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5235010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5235228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5235622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5235789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5236247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5236430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5236732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmz5uo9wa 2022-11-23T02:51:39.5236998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmz5uo9wa/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5237216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5237465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.5237704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:39.5237936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:39.5238168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:39.5238606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:39.5239035Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:39.5239459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:39.5239935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:39.5240765Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5240862Z warnings.warn( 2022-11-23T02:51:39.5241675Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5241775Z warnings.warn( 2022-11-23T02:51:39.5242588Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5242686Z warnings.warn( 2022-11-23T02:51:39.5243481Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:51:39.5243575Z warnings.warn( 2022-11-23T02:51:39.5243660Z ok (4.419s) 2022-11-23T02:51:39.5243668Z 2022-11-23T02:51:39.5243951Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5244052Z Ran 1 test in 4.419s 2022-11-23T02:51:39.5244060Z 2022-11-23T02:51:39.5244137Z OK 2022-11-23T02:51:39.5244143Z 2022-11-23T02:51:39.5244253Z Generating XML reports... 2022-11-23T02:51:39.5244725Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024627.xml 2022-11-23T02:51:39.5245121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5245290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5245696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5245880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5246129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_ogo0gs 2022-11-23T02:51:39.5246389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_ogo0gs/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5246770Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5246777Z 2022-11-23T02:51:39.5246868Z Running tests... 2022-11-23T02:51:39.5247150Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5247482Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47798 2022-11-23T02:51:39.5247695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47799 2022-11-23T02:51:39.5247908Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47800 2022-11-23T02:51:39.5248116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47801 2022-11-23T02:51:39.5248511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5248682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5249090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5249271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5249573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxh0xp2_w 2022-11-23T02:51:39.5249852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxh0xp2_w/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5250079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5250473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5250643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5251048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5251237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5251491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2crxu5_ 2022-11-23T02:51:39.5251760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2crxu5_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5252152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5252312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5252714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5252897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5253148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_l9863sj 2022-11-23T02:51:39.5253416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_l9863sj/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5253642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5253863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5254258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5254424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5254829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5255010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5255263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6cobq7h 2022-11-23T02:51:39.5255583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6cobq7h/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5255809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5255898Z ok (4.370s) 2022-11-23T02:51:39.5255905Z 2022-11-23T02:51:39.5256197Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5256295Z Ran 1 test in 4.371s 2022-11-23T02:51:39.5256302Z 2022-11-23T02:51:39.5256378Z OK 2022-11-23T02:51:39.5256385Z 2022-11-23T02:51:39.5256495Z Generating XML reports... 2022-11-23T02:51:39.5256959Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024634.xml 2022-11-23T02:51:39.5257350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5257512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5257916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5258103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5258355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq__m7zy 2022-11-23T02:51:39.5258664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq__m7zy/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5259002Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5259009Z 2022-11-23T02:51:39.5259101Z Running tests... 2022-11-23T02:51:39.5259379Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5259708Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48105 2022-11-23T02:51:39.5259922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48106 2022-11-23T02:51:39.5260139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48107 2022-11-23T02:51:39.5260350Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48108 2022-11-23T02:51:39.5260747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5260915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5261317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5261501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5261750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsh_nciz 2022-11-23T02:51:39.5262019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsh_nciz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5262407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5262578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5262979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5263162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5263409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph02eonkh 2022-11-23T02:51:39.5263678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph02eonkh/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5263905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5264127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5264524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5264739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5265145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5265329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5265581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8z5nr99l 2022-11-23T02:51:39.5265849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8z5nr99l/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5266071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5266461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5266628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5267035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5267216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5267465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj967rau8 2022-11-23T02:51:39.5267774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj967rau8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5268002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5268090Z ok (4.389s) 2022-11-23T02:51:39.5268097Z 2022-11-23T02:51:39.5268379Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5268476Z Ran 1 test in 4.390s 2022-11-23T02:51:39.5268483Z 2022-11-23T02:51:39.5268551Z OK 2022-11-23T02:51:39.5268557Z 2022-11-23T02:51:39.5268668Z Generating XML reports... 2022-11-23T02:51:39.5269132Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024642.xml 2022-11-23T02:51:39.5269527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5269696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5270107Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5270288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5270541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar0k500t 2022-11-23T02:51:39.5270808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar0k500t/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5271135Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5271142Z 2022-11-23T02:51:39.5271235Z Running tests... 2022-11-23T02:51:39.5271516Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5271851Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48412 2022-11-23T02:51:39.5272066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48413 2022-11-23T02:51:39.5272279Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48414 2022-11-23T02:51:39.5272489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48415 2022-11-23T02:51:39.5272883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5273048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5273451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5273678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5273933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2fgs0ce 2022-11-23T02:51:39.5274196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2fgs0ce/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5274593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5274762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5275167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5275350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5275603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3gymtkj 2022-11-23T02:51:39.5275872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3gymtkj/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5276155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5276376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5276825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5276995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5277404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5277587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5277842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ye803vf 2022-11-23T02:51:39.5278109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ye803vf/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5278338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5278729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5278895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5279300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5279485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5279736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbm_uy6ss 2022-11-23T02:51:39.5280003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbm_uy6ss/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5280220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5280307Z ok (4.430s) 2022-11-23T02:51:39.5280317Z 2022-11-23T02:51:39.5280599Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5280697Z Ran 1 test in 4.430s 2022-11-23T02:51:39.5280704Z 2022-11-23T02:51:39.5280780Z OK 2022-11-23T02:51:39.5280786Z 2022-11-23T02:51:39.5280894Z Generating XML reports... 2022-11-23T02:51:39.5281367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024649.xml 2022-11-23T02:51:39.5281759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5281927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5282331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5282515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5282766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe__t9yxz 2022-11-23T02:51:39.5283083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe__t9yxz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5283410Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5283418Z 2022-11-23T02:51:39.5283515Z Running tests... 2022-11-23T02:51:39.5283794Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5284123Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48719 2022-11-23T02:51:39.5284337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48720 2022-11-23T02:51:39.5284547Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48721 2022-11-23T02:51:39.5284756Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48722 2022-11-23T02:51:39.5285151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5285314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5285716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5286112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5286368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07fydwxh 2022-11-23T02:51:39.5286639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07fydwxh/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5286866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5287261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5287431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5287838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5288020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5288281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5123h6a 2022-11-23T02:51:39.5288554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5123h6a/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5288781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5289173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5289338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5289743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5289933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5290189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq03ar9v 2022-11-23T02:51:39.5290458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq03ar9v/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5290685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5291079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5291248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5291644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5291827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5292079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmw7l59b 2022-11-23T02:51:39.5292389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmw7l59b/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5292611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5292697Z ok (4.668s) 2022-11-23T02:51:39.5292708Z 2022-11-23T02:51:39.5292993Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5293093Z Ran 1 test in 4.669s 2022-11-23T02:51:39.5293100Z 2022-11-23T02:51:39.5293179Z OK 2022-11-23T02:51:39.5293185Z 2022-11-23T02:51:39.5293293Z Generating XML reports... 2022-11-23T02:51:39.5293762Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024657.xml 2022-11-23T02:51:39.5294155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5294327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5294729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5294911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5295231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbqvmyiz 2022-11-23T02:51:39.5295506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbqvmyiz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5295839Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5295846Z 2022-11-23T02:51:39.5295939Z Running tests... 2022-11-23T02:51:39.5296219Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5296529Z test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49050 2022-11-23T02:51:39.5296743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49051 2022-11-23T02:51:39.5296955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49052 2022-11-23T02:51:39.5297165Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49053 2022-11-23T02:51:39.5297562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5297732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5298135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5298317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5298572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdksnbpdh 2022-11-23T02:51:39.5298847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdksnbpdh/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5299237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5299403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5299811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5299993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5300243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptl2vi60z 2022-11-23T02:51:39.5300510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptl2vi60z/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5300902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5301069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5301521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5301703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5301956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar3c9wlk 2022-11-23T02:51:39.5302224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar3c9wlk/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5302454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5302671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5302894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5303282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5303453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5303856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5304039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5304337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzq1c6j5 2022-11-23T02:51:39.5304607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzq1c6j5/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5304829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5304916Z ok (4.554s) 2022-11-23T02:51:39.5304923Z 2022-11-23T02:51:39.5305206Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5305303Z Ran 1 test in 4.554s 2022-11-23T02:51:39.5305310Z 2022-11-23T02:51:39.5305385Z OK 2022-11-23T02:51:39.5305391Z 2022-11-23T02:51:39.5305503Z Generating XML reports... 2022-11-23T02:51:39.5305976Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024704.xml 2022-11-23T02:51:39.5306370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5306544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5306949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5307132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5307390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9e5ixep 2022-11-23T02:51:39.5307658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9e5ixep/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5307980Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5307998Z 2022-11-23T02:51:39.5308084Z Running tests... 2022-11-23T02:51:39.5308364Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5308684Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49381 2022-11-23T02:51:39.5308899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49382 2022-11-23T02:51:39.5309112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49383 2022-11-23T02:51:39.5309320Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49384 2022-11-23T02:51:39.5309714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5309879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5310285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5310514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5310766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6ca5unn 2022-11-23T02:51:39.5311040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6ca5unn/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5311266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5311657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5311823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5312227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5312409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5312662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp933wee9x 2022-11-23T02:51:39.5312929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp933wee9x/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5313151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5313576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5313743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5314149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5314332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5314584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyerercr 2022-11-23T02:51:39.5314854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyerercr/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5315085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5315470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5315638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5316038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5316285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5316541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpleh0_7z3 2022-11-23T02:51:39.5316808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpleh0_7z3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5317031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5317122Z ok (6.561s) 2022-11-23T02:51:39.5317129Z 2022-11-23T02:51:39.5317411Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5317509Z Ran 1 test in 6.562s 2022-11-23T02:51:39.5317516Z 2022-11-23T02:51:39.5317593Z OK 2022-11-23T02:51:39.5317599Z 2022-11-23T02:51:39.5317712Z Generating XML reports... 2022-11-23T02:51:39.5318179Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024712.xml 2022-11-23T02:51:39.5318573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5318741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5319139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5319326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5319642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpges0hdkl 2022-11-23T02:51:39.5319912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpges0hdkl/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5320251Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5320259Z 2022-11-23T02:51:39.5320351Z Running tests... 2022-11-23T02:51:39.5320628Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5320944Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49712 2022-11-23T02:51:39.5321159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49713 2022-11-23T02:51:39.5321370Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49714 2022-11-23T02:51:39.5321578Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49715 2022-11-23T02:51:39.5321973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5322139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5322594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5322778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5323032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2yaukr45 2022-11-23T02:51:39.5323299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2yaukr45/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5323522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5323915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5324085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5324493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5324670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5324922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8ir17ck 2022-11-23T02:51:39.5325192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8ir17ck/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5325413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5325804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5325973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5326374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5326559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5326809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rsovu9v 2022-11-23T02:51:39.5327081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rsovu9v/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5327304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5327694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5327861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5328265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5328446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5328748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyic1mn2l 2022-11-23T02:51:39.5329012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyic1mn2l/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5329240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5329326Z ok (4.546s) 2022-11-23T02:51:39.5329333Z 2022-11-23T02:51:39.5329617Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5329712Z Ran 1 test in 4.546s 2022-11-23T02:51:39.5329719Z 2022-11-23T02:51:39.5329787Z OK 2022-11-23T02:51:39.5329800Z 2022-11-23T02:51:39.5329901Z Generating XML reports... 2022-11-23T02:51:39.5330367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024722.xml 2022-11-23T02:51:39.5330757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5330929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5331333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5331561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5331817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnu5gfde 2022-11-23T02:51:39.5332088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnu5gfde/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5332420Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5332426Z 2022-11-23T02:51:39.5332521Z Running tests... 2022-11-23T02:51:39.5332799Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5333111Z test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50019 2022-11-23T02:51:39.5333326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50020 2022-11-23T02:51:39.5333537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50021 2022-11-23T02:51:39.5333747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50022 2022-11-23T02:51:39.5334143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5334312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5334718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5334904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5335156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksde7xbk 2022-11-23T02:51:39.5335428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksde7xbk/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5335645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5336040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5336209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5336608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5336789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5337040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsa9ch_4k 2022-11-23T02:51:39.5337305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsa9ch_4k/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5337690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5337905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5338311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5338496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5338746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp728rxt3k 2022-11-23T02:51:39.5339012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp728rxt3k/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5339241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5339462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5339853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5340023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5340421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5340675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5340928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_l4x3fv 2022-11-23T02:51:39.5341191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_l4x3fv/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5341406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5341491Z ok (4.740s) 2022-11-23T02:51:39.5341498Z 2022-11-23T02:51:39.5341782Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5341881Z Ran 1 test in 4.740s 2022-11-23T02:51:39.5341888Z 2022-11-23T02:51:39.5341971Z OK 2022-11-23T02:51:39.5341977Z 2022-11-23T02:51:39.5342089Z Generating XML reports... 2022-11-23T02:51:39.5342555Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024729.xml 2022-11-23T02:51:39.5342950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5343116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5343518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5343702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5343957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3y7mioj_ 2022-11-23T02:51:39.5344223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3y7mioj_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5344553Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5344560Z 2022-11-23T02:51:39.5344655Z Running tests... 2022-11-23T02:51:39.5344937Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5345259Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50326 2022-11-23T02:51:39.5345475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50327 2022-11-23T02:51:39.5345684Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50328 2022-11-23T02:51:39.5345894Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50329 2022-11-23T02:51:39.5346286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5346454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5346905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5347088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5347343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0y13ljbr 2022-11-23T02:51:39.5347615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0y13ljbr/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5347840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5348228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5348398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5348799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5348985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5349234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_cozyca 2022-11-23T02:51:39.5349500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_cozyca/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5349935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5350108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5350512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5350696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5350950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiuri250k 2022-11-23T02:51:39.5351218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiuri250k/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5351449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5351674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5352076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5352244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5352637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5352822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5353071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy61z_12i 2022-11-23T02:51:39.5353334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy61z_12i/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5353559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5353646Z ok (5.052s) 2022-11-23T02:51:39.5353653Z 2022-11-23T02:51:39.5353934Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5354030Z Ran 1 test in 5.053s 2022-11-23T02:51:39.5354037Z 2022-11-23T02:51:39.5354115Z OK 2022-11-23T02:51:39.5354122Z 2022-11-23T02:51:39.5354230Z Generating XML reports... 2022-11-23T02:51:39.5354693Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024737.xml 2022-11-23T02:51:39.5355082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5355253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5355657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5355885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5356193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bdkb944 2022-11-23T02:51:39.5356467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bdkb944/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5356804Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5356811Z 2022-11-23T02:51:39.5356906Z Running tests... 2022-11-23T02:51:39.5357189Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5357504Z test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50633 2022-11-23T02:51:39.5357720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50634 2022-11-23T02:51:39.5357924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50635 2022-11-23T02:51:39.5358137Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50636 2022-11-23T02:51:39.5358531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5358701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5359160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5359347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5359602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakk1au0n 2022-11-23T02:51:39.5359871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakk1au0n/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5360092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5360486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5360658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5361061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5361247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5361500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfocbn70z 2022-11-23T02:51:39.5361769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfocbn70z/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5361996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5362390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5362557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5362966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5363149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5363404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpta4gc4j1 2022-11-23T02:51:39.5363668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpta4gc4j1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5363892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5364286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5364455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5364859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5365119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5365371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmpsto46 2022-11-23T02:51:39.5365639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmpsto46/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5365868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5365957Z ok (4.447s) 2022-11-23T02:51:39.5365964Z 2022-11-23T02:51:39.5366250Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5366347Z Ran 1 test in 4.448s 2022-11-23T02:51:39.5366353Z 2022-11-23T02:51:39.5366429Z OK 2022-11-23T02:51:39.5366436Z 2022-11-23T02:51:39.5366551Z Generating XML reports... 2022-11-23T02:51:39.5367025Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024745.xml 2022-11-23T02:51:39.5367417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5367587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5367992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5368222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5368481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudjaozlk 2022-11-23T02:51:39.5368752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudjaozlk/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5369077Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5369096Z 2022-11-23T02:51:39.5369182Z Running tests... 2022-11-23T02:51:39.5369466Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5369784Z test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50940 2022-11-23T02:51:39.5369999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50941 2022-11-23T02:51:39.5370210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50942 2022-11-23T02:51:39.5370418Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50943 2022-11-23T02:51:39.5370812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5370977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5371383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5371565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5371818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqhw96xn 2022-11-23T02:51:39.5372089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqhw96xn/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5372480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5372645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5373051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5373232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5373481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5m3fdvj6 2022-11-23T02:51:39.5373753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5m3fdvj6/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5374148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5374359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5374765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5374943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5375194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprizwqgf2 2022-11-23T02:51:39.5375419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5375688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprizwqgf2/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5375912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5376131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5376526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5376692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5377097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5377325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5377581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0sx6ma3 2022-11-23T02:51:39.5377850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0sx6ma3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5378072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5378159Z ok (4.344s) 2022-11-23T02:51:39.5378166Z 2022-11-23T02:51:39.5378449Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5378551Z Ran 1 test in 4.345s 2022-11-23T02:51:39.5378558Z 2022-11-23T02:51:39.5378636Z OK 2022-11-23T02:51:39.5378643Z 2022-11-23T02:51:39.5378752Z Generating XML reports... 2022-11-23T02:51:39.5379216Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024753.xml 2022-11-23T02:51:39.5379612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5379778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5380173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5380354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5380608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4m2lyw7 2022-11-23T02:51:39.5380875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4m2lyw7/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5381205Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5381212Z 2022-11-23T02:51:39.5381306Z Running tests... 2022-11-23T02:51:39.5381583Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5381906Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51271 2022-11-23T02:51:39.5382120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51272 2022-11-23T02:51:39.5382329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51273 2022-11-23T02:51:39.5382539Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51274 2022-11-23T02:51:39.5382926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5383094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5383550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5383733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5383990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpor3kh0gw 2022-11-23T02:51:39.5384258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpor3kh0gw/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5384484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5384874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5385041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5385442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5385627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5385871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qdjoj2b 2022-11-23T02:51:39.5386347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qdjoj2b/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5386576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5386972Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5387138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5387542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5387722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5387978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8my6vtl 2022-11-23T02:51:39.5388247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8my6vtl/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5388474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5388870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5389038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5389447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5389631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5389882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9iw7_jli 2022-11-23T02:51:39.5390153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9iw7_jli/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5390382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5397255Z ok (6.738s) 2022-11-23T02:51:39.5397269Z 2022-11-23T02:51:39.5397639Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5397757Z Ran 1 test in 6.738s 2022-11-23T02:51:39.5397765Z 2022-11-23T02:51:39.5397842Z OK 2022-11-23T02:51:39.5397849Z 2022-11-23T02:51:39.5397959Z Generating XML reports... 2022-11-23T02:51:39.5398455Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024800.xml 2022-11-23T02:51:39.5398866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5399036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5399445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5399775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5400032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgn0dyuqv 2022-11-23T02:51:39.5400308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgn0dyuqv/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5400644Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5400651Z 2022-11-23T02:51:39.5400746Z Running tests... 2022-11-23T02:51:39.5401027Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5401333Z test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51602 2022-11-23T02:51:39.5401551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51603 2022-11-23T02:51:39.5401763Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51604 2022-11-23T02:51:39.5401976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51605 2022-11-23T02:51:39.5402376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5402607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5403022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5403209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5403464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfasibayf 2022-11-23T02:51:39.5403733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfasibayf/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5403952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5404352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5404519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5404927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5405109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5405362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5873ja2 2022-11-23T02:51:39.5405626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5873ja2/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5406016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5406180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5406584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5406770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5407023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bwq9oqs 2022-11-23T02:51:39.5407296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bwq9oqs/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5407521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5407747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5408148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5408316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5408717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5408951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5409205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodtb3saw 2022-11-23T02:51:39.5409477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodtb3saw/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5409702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5409781Z ok (4.733s) 2022-11-23T02:51:39.5409788Z 2022-11-23T02:51:39.5410076Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5410173Z Ran 1 test in 4.734s 2022-11-23T02:51:39.5410180Z 2022-11-23T02:51:39.5410256Z OK 2022-11-23T02:51:39.5410262Z 2022-11-23T02:51:39.5410373Z Generating XML reports... 2022-11-23T02:51:39.5410841Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024810.xml 2022-11-23T02:51:39.5411239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5411406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5411860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5412049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5412299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57x79lyy 2022-11-23T02:51:39.5412565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57x79lyy/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5412894Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5412901Z 2022-11-23T02:51:39.5412994Z Running tests... 2022-11-23T02:51:39.5413272Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5413584Z test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51909 2022-11-23T02:51:39.5413801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51910 2022-11-23T02:51:39.5414015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51911 2022-11-23T02:51:39.5414224Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51912 2022-11-23T02:51:39.5414618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5414785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5415185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5415370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5415625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpna_1jno1 2022-11-23T02:51:39.5415891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpna_1jno1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5416114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5416512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5416677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5417084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5417268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5417521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcel6yh6e 2022-11-23T02:51:39.5417790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcel6yh6e/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5418240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5418407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5418814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5418998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5419250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0c9kspq 2022-11-23T02:51:39.5419515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0c9kspq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5419743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5419959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5420357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5420525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5420981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5421162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5421413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufoi1sz5 2022-11-23T02:51:39.5421678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufoi1sz5/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5421901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5421988Z ok (4.382s) 2022-11-23T02:51:39.5421996Z 2022-11-23T02:51:39.5422282Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5422383Z Ran 1 test in 4.383s 2022-11-23T02:51:39.5422391Z 2022-11-23T02:51:39.5422466Z OK 2022-11-23T02:51:39.5422473Z 2022-11-23T02:51:39.5422580Z Generating XML reports... 2022-11-23T02:51:39.5423049Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024818.xml 2022-11-23T02:51:39.5423443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5423609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5424011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5424192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5424447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5g5q08vv 2022-11-23T02:51:39.5424718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5g5q08vv/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5425048Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5425055Z 2022-11-23T02:51:39.5425146Z Running tests... 2022-11-23T02:51:39.5425426Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5425740Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52216 2022-11-23T02:51:39.5425949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52217 2022-11-23T02:51:39.5426153Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52218 2022-11-23T02:51:39.5426360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52219 2022-11-23T02:51:39.5426755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5426970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5427375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5427558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5427811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm94nlvzv 2022-11-23T02:51:39.5428081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm94nlvzv/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5428305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5428698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5428865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5429270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5429454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5429705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5ie7x6y 2022-11-23T02:51:39.5430017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5ie7x6y/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5430409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5430578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5430983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5431168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5431423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppgajx3hu 2022-11-23T02:51:39.5431694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppgajx3hu/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5431911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5432138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5432535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5432703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5433109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5433292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5433546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmmmqswm 2022-11-23T02:51:39.5433819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmmmqswm/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5434043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5434131Z ok (4.681s) 2022-11-23T02:51:39.5434138Z 2022-11-23T02:51:39.5434421Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5434519Z Ran 1 test in 4.682s 2022-11-23T02:51:39.5434527Z 2022-11-23T02:51:39.5434603Z OK 2022-11-23T02:51:39.5434609Z 2022-11-23T02:51:39.5434719Z Generating XML reports... 2022-11-23T02:51:39.5435184Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024825.xml 2022-11-23T02:51:39.5435579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5435745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5436291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5436478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5436731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_izjqx_e 2022-11-23T02:51:39.5437006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_izjqx_e/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5437325Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5437340Z 2022-11-23T02:51:39.5437425Z Running tests... 2022-11-23T02:51:39.5437706Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5438014Z test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52523 2022-11-23T02:51:39.5438225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52524 2022-11-23T02:51:39.5438439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52525 2022-11-23T02:51:39.5438646Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52526 2022-11-23T02:51:39.5439114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5439283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5439685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5439867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5440120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpteqlf5k8 2022-11-23T02:51:39.5440391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpteqlf5k8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5440616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5441010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5441175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5441582Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5441761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5442010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yvtcwok 2022-11-23T02:51:39.5442275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yvtcwok/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5442499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5442891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5443053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5443459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5443647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5443902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy72ml8b9 2022-11-23T02:51:39.5444168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy72ml8b9/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5444394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5444781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5444944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5445401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5445580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5445832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu09bx3ck 2022-11-23T02:51:39.5446102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu09bx3ck/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5446328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5446415Z ok (4.557s) 2022-11-23T02:51:39.5446422Z 2022-11-23T02:51:39.5446705Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5446802Z Ran 1 test in 4.558s 2022-11-23T02:51:39.5446810Z 2022-11-23T02:51:39.5446886Z OK 2022-11-23T02:51:39.5446892Z 2022-11-23T02:51:39.5447002Z Generating XML reports... 2022-11-23T02:51:39.5447472Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024833.xml 2022-11-23T02:51:39.5447866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5448032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5448471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5448657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5448913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ww1kf2p 2022-11-23T02:51:39.5449183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ww1kf2p/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5449510Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5449517Z 2022-11-23T02:51:39.5449608Z Running tests... 2022-11-23T02:51:39.5449888Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5450218Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52830 2022-11-23T02:51:39.5450431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52831 2022-11-23T02:51:39.5450642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52832 2022-11-23T02:51:39.5450850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52833 2022-11-23T02:51:39.5451246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5451412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5451815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5452002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5452256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglvykfsh 2022-11-23T02:51:39.5452526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglvykfsh/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5452753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5453145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5453310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5453713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5453896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5454144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfds1dtpw 2022-11-23T02:51:39.5454461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfds1dtpw/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5454687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5455086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5455253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5455658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5455838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5456089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl40nu_39 2022-11-23T02:51:39.5456351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl40nu_39/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5456576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5456968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5457136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5457585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5457766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5458017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohy4zmoj 2022-11-23T02:51:39.5458285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohy4zmoj/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5458506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5458592Z ok (4.252s) 2022-11-23T02:51:39.5458602Z 2022-11-23T02:51:39.5458885Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5458983Z Ran 1 test in 4.253s 2022-11-23T02:51:39.5458990Z 2022-11-23T02:51:39.5459065Z OK 2022-11-23T02:51:39.5459071Z 2022-11-23T02:51:39.5459173Z Generating XML reports... 2022-11-23T02:51:39.5459642Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024840.xml 2022-11-23T02:51:39.5460034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5460199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5460603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5460784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5461035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxl1fbddz 2022-11-23T02:51:39.5461311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxl1fbddz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5461644Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5461651Z 2022-11-23T02:51:39.5461744Z Running tests... 2022-11-23T02:51:39.5462026Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5462335Z test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53137 2022-11-23T02:51:39.5462552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53138 2022-11-23T02:51:39.5462763Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53139 2022-11-23T02:51:39.5462969Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53140 2022-11-23T02:51:39.5463363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5463576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5463982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5464168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5464420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuj3nyb6n 2022-11-23T02:51:39.5464691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuj3nyb6n/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5464914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5465301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5465464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5465869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5466052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5466349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgit0j2j 2022-11-23T02:51:39.5466618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgit0j2j/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5466843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5467237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5467404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5467812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5467998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5468251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrdol3fb 2022-11-23T02:51:39.5468520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrdol3fb/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5468909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5469073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5469476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5469660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5469915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpep3600vy 2022-11-23T02:51:39.5470179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpep3600vy/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5470408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5470629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5470708Z ok (4.713s) 2022-11-23T02:51:39.5470725Z 2022-11-23T02:51:39.5470999Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5471097Z Ran 1 test in 4.713s 2022-11-23T02:51:39.5471104Z 2022-11-23T02:51:39.5471178Z OK 2022-11-23T02:51:39.5471185Z 2022-11-23T02:51:39.5471295Z Generating XML reports... 2022-11-23T02:51:39.5471766Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024848.xml 2022-11-23T02:51:39.5472157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5472322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5472779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5472962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5473218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvo771s7m 2022-11-23T02:51:39.5473486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvo771s7m/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5473811Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5473818Z 2022-11-23T02:51:39.5473911Z Running tests... 2022-11-23T02:51:39.5474190Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5474506Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53468 2022-11-23T02:51:39.5474728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53469 2022-11-23T02:51:39.5474938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53470 2022-11-23T02:51:39.5475147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53471 2022-11-23T02:51:39.5475585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5475754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5476251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5476437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5476694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2i6svyb 2022-11-23T02:51:39.5476962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2i6svyb/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5477187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5477581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5477750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5478158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5478342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5478593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp551ie0ks 2022-11-23T02:51:39.5478859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp551ie0ks/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5479082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5479475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5479640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5480040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5480228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5480481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpmuctz0 2022-11-23T02:51:39.5480751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpmuctz0/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5480976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5481365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5481531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5481993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5482170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5482423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_fmlmgn 2022-11-23T02:51:39.5482689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_fmlmgn/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5482914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5483000Z ok (8.557s) 2022-11-23T02:51:39.5483007Z 2022-11-23T02:51:39.5483291Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5483389Z Ran 1 test in 8.558s 2022-11-23T02:51:39.5483397Z 2022-11-23T02:51:39.5483475Z OK 2022-11-23T02:51:39.5483481Z 2022-11-23T02:51:39.5483590Z Generating XML reports... 2022-11-23T02:51:39.5484060Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024855.xml 2022-11-23T02:51:39.5484455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5484675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5485083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5485266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5485518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2iep25zx 2022-11-23T02:51:39.5485789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2iep25zx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5486119Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5486130Z 2022-11-23T02:51:39.5486221Z Running tests... 2022-11-23T02:51:39.5486503Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5486825Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53799 2022-11-23T02:51:39.5487041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53800 2022-11-23T02:51:39.5487243Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53801 2022-11-23T02:51:39.5487450Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53802 2022-11-23T02:51:39.5487845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5488012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5488418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5488608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5488862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vyh0ind 2022-11-23T02:51:39.5489134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vyh0ind/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5489356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5489744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5489911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5490317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5490499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5490801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqke387r7 2022-11-23T02:51:39.5491068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqke387r7/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5491290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5491691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5491857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5492259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5492440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5492693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmr0gbb1 2022-11-23T02:51:39.5492960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmr0gbb1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5493344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5493509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5493953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5494138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5494390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxjpl2w9 2022-11-23T02:51:39.5494660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxjpl2w9/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5494884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5495107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5495197Z ok (4.540s) 2022-11-23T02:51:39.5495204Z 2022-11-23T02:51:39.5495489Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5495587Z Ran 1 test in 4.541s 2022-11-23T02:51:39.5495594Z 2022-11-23T02:51:39.5495673Z OK 2022-11-23T02:51:39.5495680Z 2022-11-23T02:51:39.5495795Z Generating XML reports... 2022-11-23T02:51:39.5496262Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024907.xml 2022-11-23T02:51:39.5496655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5496819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5497226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5497410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5497664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp306l_wu5 2022-11-23T02:51:39.5497928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp306l_wu5/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5498257Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5498267Z 2022-11-23T02:51:39.5498353Z Running tests... 2022-11-23T02:51:39.5498629Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5498936Z test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54110 2022-11-23T02:51:39.5499150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54111 2022-11-23T02:51:39.5499367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54112 2022-11-23T02:51:39.5499575Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54113 2022-11-23T02:51:39.5500023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5500191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5500599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5500781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5501032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_4quv18 2022-11-23T02:51:39.5501301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_4quv18/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5501525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5501912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5502082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5502484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5502666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5502965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9qfrogn 2022-11-23T02:51:39.5503236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9qfrogn/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5503461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5503854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5504019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5504419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5504605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5504859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph38x_vjc 2022-11-23T02:51:39.5505128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph38x_vjc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5505354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5505839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5505956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5506243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5506373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5506556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpay7dalbr 2022-11-23T02:51:39.5506747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpay7dalbr/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5506909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5506971Z ok (4.377s) 2022-11-23T02:51:39.5506976Z 2022-11-23T02:51:39.5507176Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5507245Z Ran 1 test in 4.377s 2022-11-23T02:51:39.5507250Z 2022-11-23T02:51:39.5507304Z OK 2022-11-23T02:51:39.5507308Z 2022-11-23T02:51:39.5507386Z Generating XML reports... 2022-11-23T02:51:39.5507715Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024915.xml 2022-11-23T02:51:39.5507992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5508166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5508451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5508573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5508752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ssuo2eq 2022-11-23T02:51:39.5508942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ssuo2eq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5509174Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5509179Z 2022-11-23T02:51:39.5509244Z Running tests... 2022-11-23T02:51:39.5509442Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5509667Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54417 2022-11-23T02:51:39.5509820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54418 2022-11-23T02:51:39.5509969Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54419 2022-11-23T02:51:39.5510151Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54420 2022-11-23T02:51:39.5510434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5510552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5510835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5510963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5511143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6433uf1 2022-11-23T02:51:39.5511335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6433uf1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5511612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5511729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5512016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5512147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5512324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jd9oft3 2022-11-23T02:51:39.5512509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jd9oft3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5512784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5512902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5513186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5513314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5513492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64fyu4s2 2022-11-23T02:51:39.5513679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64fyu4s2/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5513838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5514114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5514233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5514517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5514680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5514857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4otwlkys 2022-11-23T02:51:39.5515048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4otwlkys/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5515208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5515367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5515524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5515585Z ok (5.935s) 2022-11-23T02:51:39.5515590Z 2022-11-23T02:51:39.5515790Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5515859Z Ran 1 test in 5.936s 2022-11-23T02:51:39.5515864Z 2022-11-23T02:51:39.5515920Z OK 2022-11-23T02:51:39.5515925Z 2022-11-23T02:51:39.5515996Z Generating XML reports... 2022-11-23T02:51:39.5516384Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024922.xml 2022-11-23T02:51:39.5516706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5516826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5517109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5517239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5517418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpyowb4f 2022-11-23T02:51:39.5517609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpyowb4f/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5517840Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5517848Z 2022-11-23T02:51:39.5517913Z Running tests... 2022-11-23T02:51:39.5518112Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5518330Z test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54724 2022-11-23T02:51:39.5518482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54725 2022-11-23T02:51:39.5518631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54726 2022-11-23T02:51:39.5518778Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54727 2022-11-23T02:51:39.5519057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5519175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5519461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5519594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5519777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg5pt6jdq 2022-11-23T02:51:39.5519971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg5pt6jdq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5520248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5520360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5520644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5520771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5520946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ip48li1 2022-11-23T02:51:39.5521200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ip48li1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5521477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5521598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5521882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5522009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5522188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxg81tuqf 2022-11-23T02:51:39.5522377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxg81tuqf/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5522537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5522697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5522854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5523173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5523294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5523582Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5523711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5523890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55ewiho8 2022-11-23T02:51:39.5524081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55ewiho8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5524241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5524298Z ok (4.350s) 2022-11-23T02:51:39.5524310Z 2022-11-23T02:51:39.5524502Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5524572Z Ran 1 test in 4.350s 2022-11-23T02:51:39.5524577Z 2022-11-23T02:51:39.5524632Z OK 2022-11-23T02:51:39.5524639Z 2022-11-23T02:51:39.5524717Z Generating XML reports... 2022-11-23T02:51:39.5525048Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024931.xml 2022-11-23T02:51:39.5525327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5525445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5525729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5525859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5526038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4c5e7zy3 2022-11-23T02:51:39.5526226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4c5e7zy3/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5526459Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5526464Z 2022-11-23T02:51:39.5526530Z Running tests... 2022-11-23T02:51:39.5526727Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5526948Z test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55031 2022-11-23T02:51:39.5527100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55032 2022-11-23T02:51:39.5527248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55033 2022-11-23T02:51:39.5527435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55034 2022-11-23T02:51:39.5527716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5527833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5528119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5528245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5528424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpypku8sr8 2022-11-23T02:51:39.5528614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpypku8sr8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5528772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5529048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5529167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5529454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5529617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5529796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vlcp42j 2022-11-23T02:51:39.5529985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vlcp42j/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5530143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5530422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5530539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5530829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5530958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5531138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewfkfjt8 2022-11-23T02:51:39.5531329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewfkfjt8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5531489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5531764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5531881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5532164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5532290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5532466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnh_k39y 2022-11-23T02:51:39.5532655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnh_k39y/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5532816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5532878Z ok (4.540s) 2022-11-23T02:51:39.5532883Z 2022-11-23T02:51:39.5533081Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5533151Z Ran 1 test in 4.541s 2022-11-23T02:51:39.5533156Z 2022-11-23T02:51:39.5533209Z OK 2022-11-23T02:51:39.5533214Z 2022-11-23T02:51:39.5533290Z Generating XML reports... 2022-11-23T02:51:39.5533618Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024938.xml 2022-11-23T02:51:39.5533896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5534052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5534339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5534475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5534654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt605if38 2022-11-23T02:51:39.5534842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt605if38/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5535074Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5535079Z 2022-11-23T02:51:39.5535143Z Running tests... 2022-11-23T02:51:39.5535339Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5535562Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55362 2022-11-23T02:51:39.5535719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55363 2022-11-23T02:51:39.5535869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55364 2022-11-23T02:51:39.5536047Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55365 2022-11-23T02:51:39.5536325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5536442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5536729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5536859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5537039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdz9z0x4t 2022-11-23T02:51:39.5537234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdz9z0x4t/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5537510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5537630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5537914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5538045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5538225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5twtacc 2022-11-23T02:51:39.5538412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5twtacc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5538686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5538804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5539092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5539221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5539403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqm50l204 2022-11-23T02:51:39.5539594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqm50l204/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5539754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5539913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5540060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5540338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5540493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5540782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5540912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5541089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoj93xngx 2022-11-23T02:51:39.5541275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoj93xngx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5541432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5541495Z ok (10.380s) 2022-11-23T02:51:39.5541499Z 2022-11-23T02:51:39.5541701Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5541771Z Ran 1 test in 10.381s 2022-11-23T02:51:39.5541777Z 2022-11-23T02:51:39.5541833Z OK 2022-11-23T02:51:39.5541837Z 2022-11-23T02:51:39.5541914Z Generating XML reports... 2022-11-23T02:51:39.5542242Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123024946.xml 2022-11-23T02:51:39.5542553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5542673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5542960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5543091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5543270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9trwa30g 2022-11-23T02:51:39.5543459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9trwa30g/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5543692Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5543697Z 2022-11-23T02:51:39.5543764Z Running tests... 2022-11-23T02:51:39.5543954Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5544168Z test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55693 2022-11-23T02:51:39.5544320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55694 2022-11-23T02:51:39.5544469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55695 2022-11-23T02:51:39.5544616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55696 2022-11-23T02:51:39.5544893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5545010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5545298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5545425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5545604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabyvhcg6 2022-11-23T02:51:39.5545795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabyvhcg6/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5545952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5546227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5546344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5546629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5546799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5546976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8n5nbu9r 2022-11-23T02:51:39.5547165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8n5nbu9r/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5547445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5547562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5547848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5547972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5548152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahaba3dj 2022-11-23T02:51:39.5548339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahaba3dj/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5548500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5548658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5548969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5549089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5549374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5549504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5549682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbewem7no 2022-11-23T02:51:39.5549872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbewem7no/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5550032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5550454Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5550859Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5551261Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5551666Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5551729Z ok (4.735s) 2022-11-23T02:51:39.5551735Z 2022-11-23T02:51:39.5551937Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5552007Z Ran 1 test in 4.736s 2022-11-23T02:51:39.5552012Z 2022-11-23T02:51:39.5552065Z OK 2022-11-23T02:51:39.5552070Z 2022-11-23T02:51:39.5552150Z Generating XML reports... 2022-11-23T02:51:39.5552484Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025000.xml 2022-11-23T02:51:39.5552763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5552920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5553208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5553343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5553523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8qvdfed 2022-11-23T02:51:39.5553715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8qvdfed/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5553947Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5553952Z 2022-11-23T02:51:39.5554018Z Running tests... 2022-11-23T02:51:39.5554210Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5554438Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56012 2022-11-23T02:51:39.5554594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56013 2022-11-23T02:51:39.5554742Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56014 2022-11-23T02:51:39.5554938Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56015 2022-11-23T02:51:39.5555218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5555339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5555624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5555754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5555933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0donra_ 2022-11-23T02:51:39.5556179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0donra_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5556338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5556621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5556740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5557027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5557156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5557334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8armfx2n 2022-11-23T02:51:39.5557524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8armfx2n/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5557799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5557916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5558202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5558333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5558505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjr3xjq5 2022-11-23T02:51:39.5558695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjr3xjq5/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5558853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5559007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5559291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5559450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5559737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5559868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5560047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qjmb12n 2022-11-23T02:51:39.5560236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qjmb12n/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5560393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5560807Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5561253Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5561657Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5562050Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5562453Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5562847Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5563241Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5563638Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T02:51:39.5563700Z ok (4.592s) 2022-11-23T02:51:39.5563707Z 2022-11-23T02:51:39.5563910Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5563980Z Ran 1 test in 4.593s 2022-11-23T02:51:39.5563985Z 2022-11-23T02:51:39.5564039Z OK 2022-11-23T02:51:39.5564043Z 2022-11-23T02:51:39.5564120Z Generating XML reports... 2022-11-23T02:51:39.5564449Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025007.xml 2022-11-23T02:51:39.5564727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5564882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5565168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5565299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5565483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3aqtpvyb 2022-11-23T02:51:39.5565674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3aqtpvyb/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5565906Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5565911Z 2022-11-23T02:51:39.5565978Z Running tests... 2022-11-23T02:51:39.5566174Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5566391Z test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56355 2022-11-23T02:51:39.5566545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56356 2022-11-23T02:51:39.5566693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56357 2022-11-23T02:51:39.5566876Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56358 2022-11-23T02:51:39.5567155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5567270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5567553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5567684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5567864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6v408rc 2022-11-23T02:51:39.5568056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6v408rc/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5568216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5568497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5568617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5568905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5569034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5569213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs20r6ig 2022-11-23T02:51:39.5569400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs20r6ig/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5569673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5569793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5570079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5570211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5570389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp962ctxru 2022-11-23T02:51:39.5570576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp962ctxru/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5570733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5570892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5571172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5571321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5571607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5571740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5571922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsm9a2fax 2022-11-23T02:51:39.5572111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsm9a2fax/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5572268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5572332Z ok (4.334s) 2022-11-23T02:51:39.5572337Z 2022-11-23T02:51:39.5572537Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5572607Z Ran 1 test in 4.335s 2022-11-23T02:51:39.5572612Z 2022-11-23T02:51:39.5572669Z OK 2022-11-23T02:51:39.5572673Z 2022-11-23T02:51:39.5572750Z Generating XML reports... 2022-11-23T02:51:39.5573080Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025015.xml 2022-11-23T02:51:39.5573390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5573512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5573802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5573931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5574109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_kt30pq 2022-11-23T02:51:39.5574296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_kt30pq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5574529Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5574536Z 2022-11-23T02:51:39.5574603Z Running tests... 2022-11-23T02:51:39.5574801Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5575022Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56662 2022-11-23T02:51:39.5575173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56663 2022-11-23T02:51:39.5575321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56664 2022-11-23T02:51:39.5575469Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56665 2022-11-23T02:51:39.5575745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5575866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5576153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5576283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5576463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2kagikf 2022-11-23T02:51:39.5576656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2kagikf/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5576813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5577094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5577211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5577494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5577623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5577840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6m3dzdvn 2022-11-23T02:51:39.5578031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6m3dzdvn/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5578190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5578468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5578585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5578869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5578998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5579169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60q9fqv1 2022-11-23T02:51:39.5579359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60q9fqv1/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5579518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5579828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5579947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5580234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5580364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5580546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod28qg9o 2022-11-23T02:51:39.5580739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod28qg9o/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5580899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5580961Z ok (4.757s) 2022-11-23T02:51:39.5580966Z 2022-11-23T02:51:39.5581164Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5581236Z Ran 1 test in 4.758s 2022-11-23T02:51:39.5581241Z 2022-11-23T02:51:39.5581296Z OK 2022-11-23T02:51:39.5581303Z 2022-11-23T02:51:39.5581381Z Generating XML reports... 2022-11-23T02:51:39.5581709Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025023.xml 2022-11-23T02:51:39.5581986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5582102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5582384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5582518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5582698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9tchp3u 2022-11-23T02:51:39.5582883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9tchp3u/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5583119Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5583124Z 2022-11-23T02:51:39.5583191Z Running tests... 2022-11-23T02:51:39.5583386Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5583607Z test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56969 2022-11-23T02:51:39.5583757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56970 2022-11-23T02:51:39.5583906Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56971 2022-11-23T02:51:39.5584092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56972 2022-11-23T02:51:39.5584371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5584490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5584779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5584910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5585087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fhuld3r 2022-11-23T02:51:39.5585277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fhuld3r/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5585434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5585712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5585832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5586117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5586281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5586464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4letmhw 2022-11-23T02:51:39.5586653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4letmhw/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5586811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5587082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5587200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5587486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5587613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5587792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nsczlts 2022-11-23T02:51:39.5587986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nsczlts/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5588144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5588421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5588541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5588825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5588958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5589136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei18fgbq 2022-11-23T02:51:39.5589327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei18fgbq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5589487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5589548Z ok (4.427s) 2022-11-23T02:51:39.5589553Z 2022-11-23T02:51:39.5589752Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5589823Z Ran 1 test in 4.427s 2022-11-23T02:51:39.5589828Z 2022-11-23T02:51:39.5589882Z OK 2022-11-23T02:51:39.5589886Z 2022-11-23T02:51:39.5589965Z Generating XML reports... 2022-11-23T02:51:39.5590290Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025030.xml 2022-11-23T02:51:39.5590568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5590719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5591009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5591142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5591322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdh0q8x8b 2022-11-23T02:51:39.5591513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdh0q8x8b/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5591744Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5591748Z 2022-11-23T02:51:39.5591814Z Running tests... 2022-11-23T02:51:39.5592010Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5592229Z test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57276 2022-11-23T02:51:39.5592383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57277 2022-11-23T02:51:39.5592532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57278 2022-11-23T02:51:39.5592715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57279 2022-11-23T02:51:39.5592997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5593117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5593404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5593533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5593711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hr3cbwk 2022-11-23T02:51:39.5593907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hr3cbwk/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5594067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5594344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5594463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5594738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5594868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5595047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjg1x6l_ 2022-11-23T02:51:39.5595325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5595448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5595735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5595864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5596142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwaja49d 2022-11-23T02:51:39.5596334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjg1x6l_/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5596676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwaja49d/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5596838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5596992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5597276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5597442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5597729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5597861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5598041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpah3wv52s 2022-11-23T02:51:39.5598228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpah3wv52s/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5598385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5598451Z ok (5.228s) 2022-11-23T02:51:39.5598456Z 2022-11-23T02:51:39.5598657Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5598720Z Ran 1 test in 5.228s 2022-11-23T02:51:39.5598734Z 2022-11-23T02:51:39.5598782Z OK 2022-11-23T02:51:39.5598787Z 2022-11-23T02:51:39.5598865Z Generating XML reports... 2022-11-23T02:51:39.5599194Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025038.xml 2022-11-23T02:51:39.5599510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5599630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5599914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5600043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5600223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0lswltt 2022-11-23T02:51:39.5600414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0lswltt/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5600649Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5600654Z 2022-11-23T02:51:39.5600721Z Running tests... 2022-11-23T02:51:39.5600918Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5601144Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.000s) 2022-11-23T02:51:39.5601150Z 2022-11-23T02:51:39.5601344Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5601416Z Ran 1 test in 0.001s 2022-11-23T02:51:39.5601420Z 2022-11-23T02:51:39.5601485Z OK (skipped=1) 2022-11-23T02:51:39.5601489Z 2022-11-23T02:51:39.5601566Z Generating XML reports... 2022-11-23T02:51:39.5601893Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025046.xml 2022-11-23T02:51:39.5602172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5602294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5602579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5602710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5602882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu80e_03q 2022-11-23T02:51:39.5603071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu80e_03q/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5603303Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5603308Z 2022-11-23T02:51:39.5603374Z Running tests... 2022-11-23T02:51:39.5603570Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5603790Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 621 2022-11-23T02:51:39.5603993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 622 2022-11-23T02:51:39.5604137Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 623 2022-11-23T02:51:39.5604284Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 624 2022-11-23T02:51:39.5604564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5604684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5604971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5605103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5605284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkpedhuj 2022-11-23T02:51:39.5605476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkpedhuj/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5605753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5605905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5606189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5606317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5606498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqgwcypx 2022-11-23T02:51:39.5606690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqgwcypx/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5606849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5607003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5607280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5607398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5607685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5607815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5607991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplan_hnrz 2022-11-23T02:51:39.5608179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplan_hnrz/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5608336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5608607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5608727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5609012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5609143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5609322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbus6if_g 2022-11-23T02:51:39.5609516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbus6if_g/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5609672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5609734Z ok (4.541s) 2022-11-23T02:51:39.5609739Z 2022-11-23T02:51:39.5609937Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5610040Z Ran 1 test in 4.541s 2022-11-23T02:51:39.5610045Z 2022-11-23T02:51:39.5610099Z OK 2022-11-23T02:51:39.5610103Z 2022-11-23T02:51:39.5610180Z Generating XML reports... 2022-11-23T02:51:39.5610513Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025049.xml 2022-11-23T02:51:39.5610786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5610903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5611188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5611319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5611500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpur73ryw5 2022-11-23T02:51:39.5611687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpur73ryw5/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5611919Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5611924Z 2022-11-23T02:51:39.5611989Z Running tests... 2022-11-23T02:51:39.5612185Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5612413Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s) 2022-11-23T02:51:39.5612418Z 2022-11-23T02:51:39.5612614Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5612684Z Ran 1 test in 0.000s 2022-11-23T02:51:39.5612688Z 2022-11-23T02:51:39.5612752Z OK (skipped=1) 2022-11-23T02:51:39.5612757Z 2022-11-23T02:51:39.5612834Z Generating XML reports... 2022-11-23T02:51:39.5613162Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025056.xml 2022-11-23T02:51:39.5613438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5613557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5613845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5613976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5614155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngzx47pg 2022-11-23T02:51:39.5614346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngzx47pg/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5614580Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5614585Z 2022-11-23T02:51:39.5614650Z Running tests... 2022-11-23T02:51:39.5614839Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5615071Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 989 2022-11-23T02:51:39.5615222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 990 2022-11-23T02:51:39.5615369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 991 2022-11-23T02:51:39.5615516Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 992 2022-11-23T02:51:39.5615797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5615915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5616197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5616326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5616504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkx9_ji7 2022-11-23T02:51:39.5616735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkx9_ji7/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5617014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5617134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5617415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5617544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5617721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppblvn17g 2022-11-23T02:51:39.5617911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppblvn17g/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5618070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5618231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5618508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5618627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5618938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5619072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5619251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_7n0lup 2022-11-23T02:51:39.5619439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_7n0lup/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5619596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5619873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5619993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5620276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5620407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5620586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpst58j_8o 2022-11-23T02:51:39.5620773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpst58j_8o/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5620929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5620991Z ok (5.922s) 2022-11-23T02:51:39.5620995Z 2022-11-23T02:51:39.5621194Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5621262Z Ran 1 test in 5.923s 2022-11-23T02:51:39.5621270Z 2022-11-23T02:51:39.5621325Z OK 2022-11-23T02:51:39.5621329Z 2022-11-23T02:51:39.5621406Z Generating XML reports... 2022-11-23T02:51:39.5621734Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025059.xml 2022-11-23T02:51:39.5622011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5622129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5622412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5622534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5622713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuexun0du 2022-11-23T02:51:39.5622904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuexun0du/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5623174Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5623179Z 2022-11-23T02:51:39.5623245Z Running tests... 2022-11-23T02:51:39.5623442Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5623672Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1628 2022-11-23T02:51:39.5623823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1629 2022-11-23T02:51:39.5623973Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1630 2022-11-23T02:51:39.5624118Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1631 2022-11-23T02:51:39.5624393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5624510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5624797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5624928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5625144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20iv4__7 2022-11-23T02:51:39.5625333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20iv4__7/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5625491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:39.5625768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5625886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5626170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5626302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5626481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoia2ibjo 2022-11-23T02:51:39.5626666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoia2ibjo/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5626941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5627059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5627344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5627472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5627648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpge_odx2q 2022-11-23T02:51:39.5627832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpge_odx2q/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5627994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:39.5628153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:39.5628431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5628546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5628831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5628960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5629139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4wz5b2q 2022-11-23T02:51:39.5629328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4wz5b2q/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5629521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:39.5629582Z ok (4.335s) 2022-11-23T02:51:39.5629587Z 2022-11-23T02:51:39.5629786Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5629855Z Ran 1 test in 4.336s 2022-11-23T02:51:39.5629863Z 2022-11-23T02:51:39.5629918Z OK 2022-11-23T02:51:39.5629922Z 2022-11-23T02:51:39.5629999Z Generating XML reports... 2022-11-23T02:51:39.5630321Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123025108.xml 2022-11-23T02:51:39.5630603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5630721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5631005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5631137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5631312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8s5sqcty 2022-11-23T02:51:39.5631539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8s5sqcty/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5631771Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5631776Z 2022-11-23T02:51:39.5631842Z Running tests... 2022-11-23T02:51:39.5632037Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5632156Z test_forward_backward (__main__.ReducerTest) ... ok (0.015s) 2022-11-23T02:51:39.5632160Z 2022-11-23T02:51:39.5632351Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5632421Z Ran 1 test in 0.021s 2022-11-23T02:51:39.5632426Z 2022-11-23T02:51:39.5632479Z OK 2022-11-23T02:51:39.5632486Z 2022-11-23T02:51:39.5632562Z Generating XML reports... 2022-11-23T02:51:39.5632859Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123025115.xml 2022-11-23T02:51:39.5633138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5633258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5633542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5633671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5633850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0pp6964 2022-11-23T02:51:39.5634033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0pp6964/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5634265Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5634279Z 2022-11-23T02:51:39.5634339Z Running tests... 2022-11-23T02:51:39.5634537Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5635187Z test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:51:39.5635249Z ok (0.022s) 2022-11-23T02:51:39.5635259Z 2022-11-23T02:51:39.5635448Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5635554Z Ran 1 test in 0.032s 2022-11-23T02:51:39.5635559Z 2022-11-23T02:51:39.5635615Z OK 2022-11-23T02:51:39.5635620Z 2022-11-23T02:51:39.5635696Z Generating XML reports... 2022-11-23T02:51:39.5635997Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123025118.xml 2022-11-23T02:51:39.5636350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5636471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5636757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5636888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5637069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzb69yjq 2022-11-23T02:51:39.5637261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzb69yjq/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5637497Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5637502Z 2022-11-23T02:51:39.5637569Z Running tests... 2022-11-23T02:51:39.5637769Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5637951Z test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.014s) 2022-11-23T02:51:39.5637957Z 2022-11-23T02:51:39.5638150Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5638217Z Ran 1 test in 0.022s 2022-11-23T02:51:39.5638222Z 2022-11-23T02:51:39.5638276Z OK 2022-11-23T02:51:39.5638280Z 2022-11-23T02:51:39.5638357Z Generating XML reports... 2022-11-23T02:51:39.5638650Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123025121.xml 2022-11-23T02:51:39.5638923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5639038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5639328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5639460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5639639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjd4a8mxt 2022-11-23T02:51:39.5639829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjd4a8mxt/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5640060Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5640065Z 2022-11-23T02:51:39.5640129Z Running tests... 2022-11-23T02:51:39.5640327Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5640452Z test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.008s) 2022-11-23T02:51:39.5640459Z 2022-11-23T02:51:39.5640652Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5640722Z Ran 1 test in 0.011s 2022-11-23T02:51:39.5640727Z 2022-11-23T02:51:39.5640780Z OK 2022-11-23T02:51:39.5640784Z 2022-11-23T02:51:39.5640861Z Generating XML reports... 2022-11-23T02:51:39.5641157Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123025124.xml 2022-11-23T02:51:39.5641433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5641554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5641840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5641968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5642148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzx8ccqsj 2022-11-23T02:51:39.5642379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzx8ccqsj/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5642611Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5642616Z 2022-11-23T02:51:39.5642678Z Running tests... 2022-11-23T02:51:39.5642876Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5643007Z test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.014s) 2022-11-23T02:51:39.5643012Z 2022-11-23T02:51:39.5643203Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5643271Z Ran 1 test in 0.021s 2022-11-23T02:51:39.5643276Z 2022-11-23T02:51:39.5643334Z OK 2022-11-23T02:51:39.5643338Z 2022-11-23T02:51:39.5643417Z Generating XML reports... 2022-11-23T02:51:39.5643715Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123025127.xml 2022-11-23T02:51:39.5643992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5644109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5644448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5644579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5644759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppzpvxms8 2022-11-23T02:51:39.5644951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppzpvxms8/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5645182Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5645187Z 2022-11-23T02:51:39.5645254Z Running tests... 2022-11-23T02:51:39.5645451Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5645581Z test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.008s) 2022-11-23T02:51:39.5645586Z 2022-11-23T02:51:39.5645778Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5645847Z Ran 1 test in 0.011s 2022-11-23T02:51:39.5645851Z 2022-11-23T02:51:39.5645908Z OK 2022-11-23T02:51:39.5645913Z 2022-11-23T02:51:39.5645992Z Generating XML reports... 2022-11-23T02:51:39.5646283Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123025129.xml 2022-11-23T02:51:39.5646559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5646679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5646965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5647098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5647277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3xjg80y 2022-11-23T02:51:39.5647468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3xjg80y/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5647700Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5647706Z 2022-11-23T02:51:39.5647772Z Running tests... 2022-11-23T02:51:39.5647969Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5648198Z test_logging_init (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:39.5648505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:51:39.5648566Z ok (0.498s) 2022-11-23T02:51:39.5648571Z 2022-11-23T02:51:39.5648803Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5648872Z Ran 1 test in 0.499s 2022-11-23T02:51:39.5648876Z 2022-11-23T02:51:39.5648931Z OK 2022-11-23T02:51:39.5648935Z 2022-11-23T02:51:39.5649014Z Generating XML reports... 2022-11-23T02:51:39.5649329Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221123025132.xml 2022-11-23T02:51:39.5649607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:39.5649726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:39.5650011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:39.5650135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:39.5650315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzu5mten 2022-11-23T02:51:39.5650509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzu5mten/_remote_module_non_scriptable.py 2022-11-23T02:51:39.5650742Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T02:51:39.5650747Z 2022-11-23T02:51:39.5650813Z Running tests... 2022-11-23T02:51:39.5651066Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5651747Z test_default_store_timeout_gloo (__main__.TimeoutTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.481s) 2022-11-23T02:51:39.5651752Z 2022-11-23T02:51:39.5651949Z ---------------------------------------------------------------------- 2022-11-23T02:51:39.5652017Z Ran 1 test in 0.481s 2022-11-23T02:51:39.5652024Z 2022-11-23T02:51:39.5652090Z OK (skipped=1) 2022-11-23T02:51:39.5652094Z 2022-11-23T02:51:39.5652173Z Generating XML reports... 2022-11-23T02:51:39.5652470Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221123025136.xml 2022-11-23T02:51:39.5652475Z 2022-11-23T02:51:39.5652872Z ##[endgroup] 2022-11-23T02:51:39.5653204Z FINISHED PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_gloo_eh2eef6g) 2022-11-23T02:51:39.5653209Z 2022-11-23T02:51:39.5653400Z Running distributed/test_c10d_common ... [2022-11-23 02:51:39.394175] 2022-11-23T02:51:39.5653835Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:51:39.394711] 2022-11-23T02:52:43.0323473Z 2022-11-23T02:52:43.0324341Z Expand the folded group to see the log file of distributed/test_c10d_common 2022-11-23T02:52:43.0330311Z ##[group]PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_common_bntthtce) 2022-11-23T02:52:43.0331793Z ]> 2022-11-23T02:52:43.0332801Z test_debug_level (__main__.CommTest) 2022-11-23T02:52:43.0335038Z , <__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_single_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_single_dtype>]> 2022-11-23T02:52:43.0337315Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-11-23T02:52:43.0338505Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-11-23T02:52:43.0339648Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-11-23T02:52:43.0341243Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-11-23T02:52:43.0343651Z , <__main__.PythonProcessGroupExtensionTest testMethod=test_collectives>, <__main__.PythonProcessGroupExtensionTest testMethod=test_get_backend_name>, <__main__.PythonProcessGroupExtensionTest testMethod=test_send_recv>]> 2022-11-23T02:52:43.0345959Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) 2022-11-23T02:52:43.0347118Z test_collectives (__main__.PythonProcessGroupExtensionTest) 2022-11-23T02:52:43.0348258Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) 2022-11-23T02:52:43.0349340Z test_send_recv (__main__.PythonProcessGroupExtensionTest) 2022-11-23T02:52:43.0351054Z , <__main__.ReduceOpTest testMethod=test_reduceop_copyable>, <__main__.ReduceOpTest testMethod=test_reduceop_pickle>]> 2022-11-23T02:52:43.0352631Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest) 2022-11-23T02:52:43.0353546Z test_reduceop_copyable (__main__.ReduceOpTest) 2022-11-23T02:52:43.0354411Z test_reduceop_pickle (__main__.ReduceOpTest) 2022-11-23T02:52:43.0356727Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0358526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0359715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0361333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0363496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0364405Z 2022-11-23T02:52:43.0364845Z Running tests... 2022-11-23T02:52:43.0366915Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0368263Z test_debug_level (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2567 2022-11-23T02:52:43.0369611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2568 2022-11-23T02:52:43.0371324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0372493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0374096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0375355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0376541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:52:43.0378250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0379449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0381042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0382291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0383436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:52:43.0384343Z ok (3.641s) 2022-11-23T02:52:43.0384707Z 2022-11-23T02:52:43.0385454Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0386311Z Ran 1 test in 3.642s 2022-11-23T02:52:43.0386716Z 2022-11-23T02:52:43.0386924Z OK 2022-11-23T02:52:43.0387253Z 2022-11-23T02:52:43.0387551Z Generating XML reports... 2022-11-23T02:52:43.0389046Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221123025142.xml 2022-11-23T02:52:43.0390995Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0392708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0393911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0395512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0397016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0397616Z 2022-11-23T02:52:43.0397868Z Running tests... 2022-11-23T02:52:43.0398962Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0400164Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.481s) 2022-11-23T02:52:43.0400840Z 2022-11-23T02:52:43.0401562Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0402417Z Ran 1 test in 0.482s 2022-11-23T02:52:43.0402816Z 2022-11-23T02:52:43.0403027Z OK 2022-11-23T02:52:43.0403351Z 2022-11-23T02:52:43.0403651Z Generating XML reports... 2022-11-23T02:52:43.0405401Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123025149.xml 2022-11-23T02:52:43.0407557Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0409286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0410497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0412097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0413360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0413955Z 2022-11-23T02:52:43.0414216Z Running tests... 2022-11-23T02:52:43.0415334Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0416495Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.482s) 2022-11-23T02:52:43.0417184Z 2022-11-23T02:52:43.0417904Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0418759Z Ran 1 test in 0.482s 2022-11-23T02:52:43.0419158Z 2022-11-23T02:52:43.0419370Z OK 2022-11-23T02:52:43.0419688Z 2022-11-23T02:52:43.0419985Z Generating XML reports... 2022-11-23T02:52:43.0421735Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123025152.xml 2022-11-23T02:52:43.0423603Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0425331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0426505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0428123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0429383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0429970Z 2022-11-23T02:52:43.0430234Z Running tests... 2022-11-23T02:52:43.0431350Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0432525Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.481s) 2022-11-23T02:52:43.0433210Z 2022-11-23T02:52:43.0433947Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0434769Z Ran 1 test in 0.481s 2022-11-23T02:52:43.0435163Z 2022-11-23T02:52:43.0435372Z OK 2022-11-23T02:52:43.0435696Z 2022-11-23T02:52:43.0435986Z Generating XML reports... 2022-11-23T02:52:43.0437958Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123025155.xml 2022-11-23T02:52:43.0439014Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0439698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0440192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0440819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0441340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0441585Z 2022-11-23T02:52:43.0441698Z Running tests... 2022-11-23T02:52:43.0442153Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0442639Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.492s) 2022-11-23T02:52:43.0442917Z 2022-11-23T02:52:43.0443209Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0443563Z Ran 1 test in 0.492s 2022-11-23T02:52:43.0443707Z 2022-11-23T02:52:43.0443801Z OK 2022-11-23T02:52:43.0443938Z 2022-11-23T02:52:43.0444066Z Generating XML reports... 2022-11-23T02:52:43.0444815Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123025159.xml 2022-11-23T02:52:43.0445552Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0446237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0446725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0447364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0447852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0448095Z 2022-11-23T02:52:43.0448205Z Running tests... 2022-11-23T02:52:43.0448659Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0449268Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2976 2022-11-23T02:52:43.0449874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2977 2022-11-23T02:52:43.0450360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2978 2022-11-23T02:52:43.0450834Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2979 2022-11-23T02:52:43.0451482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0451972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0452618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0453134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0453615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:52:43.0454299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0454784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0455398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0455900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0456377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:52:43.0457060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0457592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0458228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0458735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0459215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:52:43.0459873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0460363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0461000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0461503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0461980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:52:43.0462358Z ok (4.021s) 2022-11-23T02:52:43.0462516Z 2022-11-23T02:52:43.0462817Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0463150Z Ran 1 test in 4.022s 2022-11-23T02:52:43.0463316Z 2022-11-23T02:52:43.0463408Z OK 2022-11-23T02:52:43.0463543Z 2022-11-23T02:52:43.0463716Z Generating XML reports... 2022-11-23T02:52:43.0464446Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123025202.xml 2022-11-23T02:52:43.0465202Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0465892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0466385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0466996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0467506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0467746Z 2022-11-23T02:52:43.0467856Z Running tests... 2022-11-23T02:52:43.0468312Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0468912Z test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3271 2022-11-23T02:52:43.0469513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3272 2022-11-23T02:52:43.0469995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3273 2022-11-23T02:52:43.0470445Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3274 2022-11-23T02:52:43.0471100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0471589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0472237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0472735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0473220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:52:43.0473934Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-11-23T02:52:43.0474622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0475105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0475738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0476313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0476855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:52:43.0477571Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-11-23T02:52:43.0478294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0478782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0479396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0479900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0480380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:52:43.0481064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0481554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0482191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0482747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0483198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:52:43.0483734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:52:43.0484265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:52:43.0484796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:52:43.0485319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:52:43.0486060Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:52:43.0486818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:52:43.0487565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:52:43.0488294Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:52:43.0488717Z ok (5.432s) 2022-11-23T02:52:43.0488870Z 2022-11-23T02:52:43.0489171Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0489523Z Ran 1 test in 5.433s 2022-11-23T02:52:43.0489693Z 2022-11-23T02:52:43.0489785Z OK 2022-11-23T02:52:43.0489920Z 2022-11-23T02:52:43.0490045Z Generating XML reports... 2022-11-23T02:52:43.0490770Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123025209.xml 2022-11-23T02:52:43.0491511Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0492200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0492684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0493316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0493826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0494066Z 2022-11-23T02:52:43.0494175Z Running tests... 2022-11-23T02:52:43.0494628Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0495212Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3571 2022-11-23T02:52:43.0495862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3572 2022-11-23T02:52:43.0496341Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3573 2022-11-23T02:52:43.0496835Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3574 2022-11-23T02:52:43.0497507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0497992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0498635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0499142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0499597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:52:43.0500281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0500770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0501408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0501953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0502431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:52:43.0503121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0503587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0504222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0504726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0505206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:52:43.0505891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0506379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0507019Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0507499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0507971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:52:43.0508353Z ok (4.223s) 2022-11-23T02:52:43.0508511Z 2022-11-23T02:52:43.0508810Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0509169Z Ran 1 test in 4.223s 2022-11-23T02:52:43.0509338Z 2022-11-23T02:52:43.0509436Z OK 2022-11-23T02:52:43.0509570Z 2022-11-23T02:52:43.0509700Z Generating XML reports... 2022-11-23T02:52:43.0510394Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123025217.xml 2022-11-23T02:52:43.0511163Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0511847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0512331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0512967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0513477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0513718Z 2022-11-23T02:52:43.0513830Z Running tests... 2022-11-23T02:52:43.0514263Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0514906Z test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3866 2022-11-23T02:52:43.0515502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3867 2022-11-23T02:52:43.0515993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3868 2022-11-23T02:52:43.0516557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3869 2022-11-23T02:52:43.0517245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0517730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0518342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0518859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0519344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:52:43.0520062Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-11-23T02:52:43.0520874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0521356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0521997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0522512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0522971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:52:43.0523686Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-11-23T02:52:43.0524406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0524884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0525518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0526019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0526489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:52:43.0527172Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-11-23T02:52:43.0527896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0528382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0529032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0529540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0530014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:52:43.0530550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:52:43.0531088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:52:43.0531593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:52:43.0532119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:52:43.0532849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:52:43.0533671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:52:43.0534426Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:52:43.0535181Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:52:43.0535596Z ok (5.428s) 2022-11-23T02:52:43.0535749Z 2022-11-23T02:52:43.0536055Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0536391Z Ran 1 test in 5.429s 2022-11-23T02:52:43.0536563Z 2022-11-23T02:52:43.0536653Z OK 2022-11-23T02:52:43.0536794Z 2022-11-23T02:52:43.0536919Z Generating XML reports... 2022-11-23T02:52:43.0537637Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123025224.xml 2022-11-23T02:52:43.0538403Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0539095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0539582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0540249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0540757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0540998Z 2022-11-23T02:52:43.0541108Z Running tests... 2022-11-23T02:52:43.0541561Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0542005Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest) ... ok (0.481s) 2022-11-23T02:52:43.0542247Z 2022-11-23T02:52:43.0542542Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0542894Z Ran 1 test in 0.482s 2022-11-23T02:52:43.0543043Z 2022-11-23T02:52:43.0543137Z OK 2022-11-23T02:52:43.0543270Z 2022-11-23T02:52:43.0543394Z Generating XML reports... 2022-11-23T02:52:43.0544019Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123025233.xml 2022-11-23T02:52:43.0544706Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0545394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0545884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0546522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0547005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0547244Z 2022-11-23T02:52:43.0547353Z Running tests... 2022-11-23T02:52:43.0547803Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0548238Z test_reduceop_copyable (__main__.ReduceOpTest) ... ok (0.484s) 2022-11-23T02:52:43.0548470Z 2022-11-23T02:52:43.0548763Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0549110Z Ran 1 test in 0.484s 2022-11-23T02:52:43.0549272Z 2022-11-23T02:52:43.0549367Z OK 2022-11-23T02:52:43.0549479Z 2022-11-23T02:52:43.0549604Z Generating XML reports... 2022-11-23T02:52:43.0550228Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123025236.xml 2022-11-23T02:52:43.0550920Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T02:52:43.0551599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:43.0552079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:43.0552713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:43.0553274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:43.0553511Z 2022-11-23T02:52:43.0553598Z Running tests... 2022-11-23T02:52:43.0554052Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0554478Z test_reduceop_pickle (__main__.ReduceOpTest) ... ok (0.494s) 2022-11-23T02:52:43.0554703Z 2022-11-23T02:52:43.0554995Z ---------------------------------------------------------------------- 2022-11-23T02:52:43.0555356Z Ran 1 test in 0.495s 2022-11-23T02:52:43.0555524Z 2022-11-23T02:52:43.0555617Z OK 2022-11-23T02:52:43.0555758Z 2022-11-23T02:52:43.0555882Z Generating XML reports... 2022-11-23T02:52:43.0556563Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123025239.xml 2022-11-23T02:52:43.0556907Z 2022-11-23T02:52:43.0557346Z ##[endgroup] 2022-11-23T02:52:43.0557987Z FINISHED PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_common_bntthtce) 2022-11-23T02:52:43.0558339Z 2022-11-23T02:52:43.0558669Z Running distributed/pipeline/sync/test_transparency ... [2022-11-23 02:52:43.034110] 2022-11-23T02:52:43.0559453Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-11-23 02:52:43.034949] 2022-11-23T02:52:46.6168903Z 2022-11-23T02:52:46.6170023Z Expand the folded group to see the log file of distributed/pipeline/sync/test_transparency 2022-11-23T02:52:46.6172595Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_transparency_4xt85vqp) 2022-11-23T02:52:46.6174135Z ============================= test session starts ============================== 2022-11-23T02:52:46.6175831Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:52:46.6176796Z cachedir: .pytest_cache 2022-11-23T02:52:46.6178392Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:52:46.6179538Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:52:46.6180391Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:52:46.6181967Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:52:46.6216179Z collecting ... collected 1 item 2022-11-23T02:52:46.6217463Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears 2022-11-23T02:52:46.6218215Z 2022-11-23T02:52:46.6218811Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%] 2022-11-23T02:52:46.6219486Z 2022-11-23T02:52:46.6219885Z ============================== 1 passed in 0.15s =============================== 2022-11-23T02:52:46.6220412Z 2022-11-23T02:52:46.6221244Z ##[endgroup] 2022-11-23T02:52:46.6223215Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_transparency_4xt85vqp) 2022-11-23T02:52:46.6224274Z 2022-11-23T02:52:46.6225104Z Running distributed/pipeline/sync/test_pipeline ... [2022-11-23 02:52:46.617408] 2022-11-23T02:52:46.6226839Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-11-23 02:52:46.618244] 2022-11-23T02:52:49.7067061Z 2022-11-23T02:52:49.7068199Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipeline 2022-11-23T02:52:49.7070714Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_pipeline__7p13zd9) 2022-11-23T02:52:49.7072217Z ============================= test session starts ============================== 2022-11-23T02:52:49.7074505Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:52:49.7075431Z cachedir: .pytest_cache 2022-11-23T02:52:49.7077432Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:52:49.7078602Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:52:49.7079443Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:52:49.7081035Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:52:49.7082101Z collecting ... collected 1 item 2022-11-23T02:52:49.7083164Z Running 1 items in this shard: test/distributed/pipeline/sync/test_pipeline.py::test_clock_cycles 2022-11-23T02:52:49.7083871Z 2022-11-23T02:52:49.7084424Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED [100%] 2022-11-23T02:52:49.7085070Z 2022-11-23T02:52:49.7085469Z ============================== 1 passed in 0.02s =============================== 2022-11-23T02:52:49.7085976Z 2022-11-23T02:52:49.7086749Z ##[endgroup] 2022-11-23T02:52:49.7088498Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_pipeline__7p13zd9) 2022-11-23T02:52:49.7089522Z 2022-11-23T02:52:49.7090548Z Running distributed/pipeline/sync/test_phony ... [2022-11-23 02:52:49.707206] 2022-11-23T02:52:49.7092273Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-11-23 02:52:49.708052] 2022-11-23T02:52:52.7814240Z 2022-11-23T02:52:52.7815343Z Expand the folded group to see the log file of distributed/pipeline/sync/test_phony 2022-11-23T02:52:52.7817800Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_phony_dibr8adr) 2022-11-23T02:52:52.7819295Z ============================= test session starts ============================== 2022-11-23T02:52:52.7821043Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:52:52.7821989Z cachedir: .pytest_cache 2022-11-23T02:52:52.7823605Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:52:52.7824788Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:52:52.7825614Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:52:52.7827197Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:52:52.7828260Z collecting ... collected 4 items 2022-11-23T02:52:52.7830101Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function 2022-11-23T02:52:52.7831521Z 2022-11-23T02:52:52.7832063Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED [ 25%] 2022-11-23T02:52:52.7833222Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%] 2022-11-23T02:52:52.7834381Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED [ 75%] 2022-11-23T02:52:52.7835594Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%] 2022-11-23T02:52:52.7836661Z 2022-11-23T02:52:52.7837075Z ============================== 4 passed in 0.03s =============================== 2022-11-23T02:52:52.7837570Z 2022-11-23T02:52:52.7838348Z ##[endgroup] 2022-11-23T02:52:52.7840135Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_phony_dibr8adr) 2022-11-23T02:52:52.7841131Z 2022-11-23T02:52:52.7841945Z Running distributed/pipeline/sync/test_inplace ... [2022-11-23 02:52:52.782028] 2022-11-23T02:52:52.7844179Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-11-23 02:52:52.782853] 2022-11-23T02:52:55.9892497Z 2022-11-23T02:52:55.9893601Z Expand the folded group to see the log file of distributed/pipeline/sync/test_inplace 2022-11-23T02:52:55.9896096Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_inplace_dpdw5uu4) 2022-11-23T02:52:55.9897612Z ============================= test session starts ============================== 2022-11-23T02:52:55.9899301Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:52:55.9900236Z cachedir: .pytest_cache 2022-11-23T02:52:55.9901840Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:52:55.9903014Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:52:55.9903870Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:52:55.9905444Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:52:55.9906517Z collecting ... collected 3 items 2022-11-23T02:52:55.9908710Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad 2022-11-23T02:52:55.9910012Z 2022-11-23T02:52:55.9910601Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%] 2022-11-23T02:52:55.9911866Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%] 2022-11-23T02:52:55.9913102Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%] 2022-11-23T02:52:55.9913789Z 2022-11-23T02:52:55.9914213Z ========================= 1 passed, 2 xfailed in 0.19s ========================= 2022-11-23T02:52:55.9914743Z 2022-11-23T02:52:55.9915519Z ##[endgroup] 2022-11-23T02:52:55.9917523Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_inplace_dpdw5uu4) 2022-11-23T02:52:55.9918552Z 2022-11-23T02:52:55.9919444Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-11-23 02:52:55.989850] 2022-11-23T02:52:55.9921291Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-11-23 02:52:55.990678] 2022-11-23T02:52:59.8011637Z 2022-11-23T02:52:59.8012648Z Expand the folded group to see the log file of distributed/pipeline/sync/test_deferred_batch_norm 2022-11-23T02:52:59.8013849Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_2bro6muy) 2022-11-23T02:52:59.8014513Z ============================= test session starts ============================== 2022-11-23T02:52:59.8015345Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:52:59.8015719Z cachedir: .pytest_cache 2022-11-23T02:52:59.8016367Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:52:59.8016840Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:52:59.8017177Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:52:59.8017800Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:52:59.8018244Z collecting ... collected 11 items 2022-11-23T02:52:59.8020726Z Running 11 items in this shard: test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad 2022-11-23T02:52:59.8022464Z 2022-11-23T02:52:59.8022860Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [ 9%] 2022-11-23T02:52:59.8023514Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%] 2022-11-23T02:52:59.8024179Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%] 2022-11-23T02:52:59.8024800Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%] 2022-11-23T02:52:59.8025321Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%] 2022-11-23T02:52:59.8025934Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%] 2022-11-23T02:52:59.8026478Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%] 2022-11-23T02:52:59.8026994Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED [ 72%] 2022-11-23T02:52:59.8027492Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%] 2022-11-23T02:52:59.8027981Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%] 2022-11-23T02:52:59.8028459Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%] 2022-11-23T02:52:59.8028749Z 2022-11-23T02:52:59.8028919Z ============================== 11 passed in 0.76s ============================== 2022-11-23T02:52:59.8029140Z 2022-11-23T02:52:59.8029484Z ##[endgroup] 2022-11-23T02:52:59.8030286Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_2bro6muy) 2022-11-23T02:52:59.8030723Z 2022-11-23T02:52:59.8031053Z Running distributed/pipeline/sync/test_checkpoint ... [2022-11-23 02:52:59.801714] 2022-11-23T02:52:59.8031764Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-11-23 02:52:59.802541] 2022-11-23T02:53:03.4345013Z 2022-11-23T02:53:03.4346663Z Expand the folded group to see the log file of distributed/pipeline/sync/test_checkpoint 2022-11-23T02:53:03.4350195Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_checkpoint_f61i3vwf) 2022-11-23T02:53:03.4351685Z ============================= test session starts ============================== 2022-11-23T02:53:03.4353562Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:53:03.4354517Z cachedir: .pytest_cache 2022-11-23T02:53:03.4356330Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:53:03.4357511Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:53:03.4358339Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:53:03.4359908Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:53:03.4361092Z collecting ... collected 9 items 2022-11-23T02:53:03.4364910Z Running 9 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output 2022-11-23T02:53:03.4368602Z 2022-11-23T02:53:03.4369220Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%] 2022-11-23T02:53:03.4370472Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%] 2022-11-23T02:53:03.4371723Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%] 2022-11-23T02:53:03.4373026Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%] 2022-11-23T02:53:03.4374350Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%] 2022-11-23T02:53:03.4375778Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%] 2022-11-23T02:53:03.4377112Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%] 2022-11-23T02:53:03.4378569Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%] 2022-11-23T02:53:03.4379897Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%] 2022-11-23T02:53:03.4380558Z 2022-11-23T02:53:03.4380956Z ============================== 9 passed in 0.56s =============================== 2022-11-23T02:53:03.4381451Z 2022-11-23T02:53:03.4382229Z ##[endgroup] 2022-11-23T02:53:03.4384112Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_checkpoint_f61i3vwf) 2022-11-23T02:53:03.4385168Z 2022-11-23T02:53:03.4385978Z Running distributed/pipeline/sync/test_balance ... [2022-11-23 02:53:03.435101] 2022-11-23T02:53:03.4387711Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-11-23 02:53:03.435949] 2022-11-23T02:53:13.9533520Z 2022-11-23T02:53:13.9534872Z Expand the folded group to see the log file of distributed/pipeline/sync/test_balance 2022-11-23T02:53:13.9538723Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_balance_82cz3loh) 2022-11-23T02:53:13.9540217Z ============================= test session starts ============================== 2022-11-23T02:53:13.9541835Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:53:13.9542773Z cachedir: .pytest_cache 2022-11-23T02:53:13.9544593Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:53:13.9545735Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:53:13.9546573Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:53:13.9548144Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:53:13.9549192Z collecting ... collected 18 items 2022-11-23T02:53:13.9555358Z Running 18 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad 2022-11-23T02:53:13.9561817Z 2022-11-23T02:53:13.9562406Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED [ 5%] 2022-11-23T02:53:13.9563606Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%] 2022-11-23T02:53:13.9564910Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%] 2022-11-23T02:53:13.9566413Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%] 2022-11-23T02:53:13.9567665Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%] 2022-11-23T02:53:13.9568864Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%] 2022-11-23T02:53:13.9570115Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%] 2022-11-23T02:53:13.9571357Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%] 2022-11-23T02:53:13.9572538Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%] 2022-11-23T02:53:13.9573739Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%] 2022-11-23T02:53:13.9574957Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%] 2022-11-23T02:53:13.9576163Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%] 2022-11-23T02:53:13.9577392Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%] 2022-11-23T02:53:13.9578654Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%] 2022-11-23T02:53:13.9579850Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED [ 83%] 2022-11-23T02:53:13.9581006Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%] 2022-11-23T02:53:13.9582169Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%] 2022-11-23T02:53:13.9583325Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED [100%] 2022-11-23T02:53:13.9583957Z 2022-11-23T02:53:13.9584378Z ======================== 16 passed, 2 skipped in 7.08s ========================= 2022-11-23T02:53:13.9584890Z 2022-11-23T02:53:13.9585672Z ##[endgroup] 2022-11-23T02:53:13.9587542Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_balance_82cz3loh) 2022-11-23T02:53:13.9588548Z 2022-11-23T02:53:13.9589359Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-11-23 02:53:13.954135] 2022-11-23T02:53:13.9591096Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-11-23 02:53:13.954965] 2022-11-23T02:53:17.3775756Z 2022-11-23T02:53:17.3776995Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_tracker 2022-11-23T02:53:17.3779478Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_tracker_smkhudjj) 2022-11-23T02:53:17.3781462Z ============================= test session starts ============================== 2022-11-23T02:53:17.3783225Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:53:17.3784169Z cachedir: .pytest_cache 2022-11-23T02:53:17.3785748Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:53:17.3786999Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:53:17.3787865Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:53:17.3789460Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:53:17.3790530Z collecting ... collected 6 items 2022-11-23T02:53:17.3793407Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker, test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel, test/distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing 2022-11-23T02:53:17.3795598Z 2022-11-23T02:53:17.3796891Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%] 2022-11-23T02:53:17.3798259Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel PASSED [ 33%] 2022-11-23T02:53:17.3799546Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%] 2022-11-23T02:53:17.3800753Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%] 2022-11-23T02:53:17.3802053Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%] 2022-11-23T02:53:17.3803388Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%] 2022-11-23T02:53:17.3804110Z 2022-11-23T02:53:17.3804512Z ============================== 6 passed in 0.36s =============================== 2022-11-23T02:53:17.3805010Z 2022-11-23T02:53:17.3805794Z ##[endgroup] 2022-11-23T02:53:17.3807703Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_tracker_smkhudjj) 2022-11-23T02:53:17.3808786Z 2022-11-23T02:53:17.3809625Z Running distributed/pipeline/sync/skip/test_portal ... [2022-11-23 02:53:17.378192] 2022-11-23T02:53:17.3811384Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-11-23 02:53:17.378947] 2022-11-23T02:53:20.7201499Z 2022-11-23T02:53:20.7203016Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_portal 2022-11-23T02:53:20.7205735Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_portal_qt84dzca) 2022-11-23T02:53:20.7207192Z ============================= test session starts ============================== 2022-11-23T02:53:20.7208965Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:53:20.7209899Z cachedir: .pytest_cache 2022-11-23T02:53:20.7211531Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:53:20.7212736Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:53:20.7213587Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:53:20.7215182Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:53:20.7216219Z collecting ... collected 10 items 2022-11-23T02:53:20.7220052Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 2022-11-23T02:53:20.7223810Z 2022-11-23T02:53:20.7224440Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%] 2022-11-23T02:53:20.7225702Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED [ 20%] 2022-11-23T02:53:20.7226951Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%] 2022-11-23T02:53:20.7228173Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED [ 40%] 2022-11-23T02:53:20.7229533Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%] 2022-11-23T02:53:20.7230830Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%] 2022-11-23T02:53:20.7232085Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%] 2022-11-23T02:53:20.7233356Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%] 2022-11-23T02:53:20.7234637Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%] 2022-11-23T02:53:20.7235966Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%] 2022-11-23T02:53:20.7237105Z 2022-11-23T02:53:20.7237507Z ============================== 10 passed in 0.33s ============================== 2022-11-23T02:53:20.7238012Z 2022-11-23T02:53:20.7238761Z ##[endgroup] 2022-11-23T02:53:20.7240660Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_portal_qt84dzca) 2022-11-23T02:53:20.7241750Z 2022-11-23T02:53:20.7242674Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-11-23 02:53:20.720857] 2022-11-23T02:53:20.7244561Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-11-23 02:53:20.721679] 2022-11-23T02:53:23.8037421Z 2022-11-23T02:53:23.8039143Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-11-23T02:53:23.8041819Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_dln0vq8d) 2022-11-23T02:53:23.8043432Z ============================= test session starts ============================== 2022-11-23T02:53:23.8045127Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:53:23.8046078Z cachedir: .pytest_cache 2022-11-23T02:53:23.8047758Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:53:23.8048925Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:53:23.8049744Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:53:23.8051337Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:53:23.8052393Z collecting ... collected 6 items 2022-11-23T02:53:23.8055169Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace 2022-11-23T02:53:23.8057894Z 2022-11-23T02:53:23.8058511Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%] 2022-11-23T02:53:23.8059823Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%] 2022-11-23T02:53:23.8061163Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%] 2022-11-23T02:53:23.8062479Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%] 2022-11-23T02:53:23.8063849Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%] 2022-11-23T02:53:23.8065181Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%] 2022-11-23T02:53:23.8065842Z 2022-11-23T02:53:23.8066398Z ============================== 6 passed in 0.03s =============================== 2022-11-23T02:53:23.8066909Z 2022-11-23T02:53:23.8067674Z ##[endgroup] 2022-11-23T02:53:23.8069696Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_dln0vq8d) 2022-11-23T02:53:23.8070855Z 2022-11-23T02:53:23.8071638Z Running distributed/pipeline/sync/skip/test_api ... [2022-11-23 02:53:23.804274] 2022-11-23T02:53:23.8073369Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-11-23 02:53:23.805104] 2022-11-23T02:53:26.8567923Z 2022-11-23T02:53:26.8569064Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_api 2022-11-23T02:53:26.8571873Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_api_m1vgvip9) 2022-11-23T02:53:26.8573388Z ============================= test session starts ============================== 2022-11-23T02:53:26.8575131Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:53:26.8576061Z cachedir: .pytest_cache 2022-11-23T02:53:26.8577635Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:53:26.8578788Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:53:26.8579630Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:53:26.8581208Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T02:53:26.8582287Z collecting ... collected 3 items 2022-11-23T02:53:26.8583929Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr 2022-11-23T02:53:26.8585170Z 2022-11-23T02:53:26.8585758Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%] 2022-11-23T02:53:26.8586968Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED [ 66%] 2022-11-23T02:53:26.8588101Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED [100%] 2022-11-23T02:53:26.8588750Z 2022-11-23T02:53:26.8589153Z ============================== 3 passed in 0.03s =============================== 2022-11-23T02:53:26.8589654Z 2022-11-23T02:53:26.8590427Z ##[endgroup] 2022-11-23T02:53:26.8592256Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_api_m1vgvip9) 2022-11-23T02:53:26.8593774Z 2022-11-23T02:53:26.8594648Z Running distributed/optim/test_apply_optimizer_in_backward ... [2022-11-23 02:53:26.857303] 2022-11-23T02:53:26.8597356Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:53:26.858169] 2022-11-23T02:53:29.6864615Z 2022-11-23T02:53:29.6866323Z Expand the folded group to see the log file of distributed/optim/test_apply_optimizer_in_backward 2022-11-23T02:53:29.6868891Z ##[group]PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/pytorch/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_g9cf4oyx) 2022-11-23T02:53:29.6869990Z 2022-11-23T02:53:29.6870742Z ##[endgroup] 2022-11-23T02:53:29.6872975Z FINISHED PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/pytorch/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_g9cf4oyx) 2022-11-23T02:53:29.6874077Z 2022-11-23T02:53:29.6874831Z Running distributed/fsdp/test_wrap ... [2022-11-23 02:53:29.687015] 2022-11-23T02:53:29.6883499Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:53:29.687911] 2022-11-23T02:53:33.2307595Z 2022-11-23T02:53:33.2308618Z Expand the folded group to see the log file of distributed/fsdp/test_wrap 2022-11-23T02:53:33.2310769Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_wrap_9hl8cy5s) 2022-11-23T02:53:33.2312733Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap 2022-11-23T02:53:33.2313501Z 2022-11-23T02:53:33.2313739Z Running tests... 2022-11-23T02:53:33.2314797Z ---------------------------------------------------------------------- 2022-11-23T02:53:33.2315799Z test_always_wrap (__main__.TestAutoWrap) 2022-11-23T02:53:33.2317673Z Test to ensure that if `always_wrap_policy` is ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.491s) 2022-11-23T02:53:33.2320029Z test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2322520Z test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2323852Z test_auto_wrap_api (__main__.TestAutoWrap) 2022-11-23T02:53:33.2325726Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2327119Z test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap) 2022-11-23T02:53:33.2329150Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2330642Z test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap) 2022-11-23T02:53:33.2332659Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2334033Z test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap) 2022-11-23T02:53:33.2335897Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2337714Z test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap) 2022-11-23T02:53:33.2339479Z Test to ensure force-leaf modules are not wrapped. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2342152Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2345130Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2348035Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2351119Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2354063Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2357130Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2360112Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2363050Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2365689Z test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2368141Z test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2369506Z test_module_wrap_policy (__main__.TestAutoWrap) 2022-11-23T02:53:33.2371128Z Tests the ``ModuleWrapPolicy``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:53:33.2372345Z test_transformer_auto_wrap_policy (__main__.TestAutoWrap) 2022-11-23T02:53:33.2374010Z Tests the ``transformer_auto_wrap_policy``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:53:33.2376111Z test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2378194Z test_wrap_override_defaults (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2380526Z test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2382741Z test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2384090Z test_bn_always_wrapped_individually (__main__.TestFSDPWrap) 2022-11-23T02:53:33.2385939Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2387440Z test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-11-23T02:53:33.2389421Z Test that an error is raised if we attempt to wrap when submodules are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2390920Z test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-11-23T02:53:33.2392987Z Test that an error is raised if we attempt to wrap when submodules are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2394468Z test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-11-23T02:53:33.2396502Z Test that an error is raised if we attempt to wrap when submodules are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2397938Z test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-11-23T02:53:33.2399897Z Test that an error is raised if we attempt to wrap when submodules are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2402866Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2406287Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2409703Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2413078Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2416463Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2419974Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2423401Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2426782Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2430210Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2433740Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2437247Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2439915Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2441395Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2442846Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2444308Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2445763Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:33.2446935Z test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2447924Z test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:33.2448395Z 2022-11-23T02:53:33.2448698Z ---------------------------------------------------------------------- 2022-11-23T02:53:33.2449051Z Ran 47 tests in 0.558s 2022-11-23T02:53:33.2449198Z 2022-11-23T02:53:33.2449310Z OK (skipped=47) 2022-11-23T02:53:33.2449476Z 2022-11-23T02:53:33.2449606Z Generating XML reports... 2022-11-23T02:53:33.2450232Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123025331.xml 2022-11-23T02:53:33.2451016Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123025331.xml 2022-11-23T02:53:33.2451363Z 2022-11-23T02:53:33.2451725Z ##[endgroup] 2022-11-23T02:53:33.2452352Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_wrap_9hl8cy5s) 2022-11-23T02:53:33.2452705Z 2022-11-23T02:53:33.2453018Z Running distributed/fsdp/test_shard_utils ... [2022-11-23 02:53:33.231810] 2022-11-23T02:53:33.2453816Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:53:33.232679] 2022-11-23T02:53:36.1870004Z 2022-11-23T02:53:36.1871123Z Expand the folded group to see the log file of distributed/fsdp/test_shard_utils 2022-11-23T02:53:36.1873476Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_shard_utils_j8jz820d) 2022-11-23T02:53:36.1874437Z 2022-11-23T02:53:36.1875187Z ##[endgroup] 2022-11-23T02:53:36.1877655Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_shard_utils_j8jz820d) 2022-11-23T02:53:36.1878561Z 2022-11-23T02:53:36.1879351Z Running distributed/fsdp/test_fsdp_uneven ... [2022-11-23 02:53:36.187366] 2022-11-23T02:53:36.1891222Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:53:36.188265] 2022-11-23T02:53:39.6373725Z 2022-11-23T02:53:39.6374486Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_uneven 2022-11-23T02:53:39.6376766Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_uneven_19psubau) 2022-11-23T02:53:39.6379004Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven 2022-11-23T02:53:39.6379762Z 2022-11-23T02:53:39.6380023Z Running tests... 2022-11-23T02:53:39.6381182Z ---------------------------------------------------------------------- 2022-11-23T02:53:39.6382232Z test_one_iteration (__main__.TestUnevenParamShard) 2022-11-23T02:53:39.6384157Z Test FSDP with uneven divide of parameter shards. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.485s) 2022-11-23T02:53:39.6385159Z 2022-11-23T02:53:39.6385967Z ---------------------------------------------------------------------- 2022-11-23T02:53:39.6386827Z Ran 1 test in 0.489s 2022-11-23T02:53:39.6387235Z 2022-11-23T02:53:39.6387492Z OK (skipped=1) 2022-11-23T02:53:39.6387874Z 2022-11-23T02:53:39.6388169Z Generating XML reports... 2022-11-23T02:53:39.6389898Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221123025337.xml 2022-11-23T02:53:39.6390879Z 2022-11-23T02:53:39.6391641Z ##[endgroup] 2022-11-23T02:53:39.6393265Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_uneven_19psubau) 2022-11-23T02:53:39.6394197Z 2022-11-23T02:53:39.6395022Z Running distributed/fsdp/test_fsdp_tp_integration ... [2022-11-23 02:53:39.638069] 2022-11-23T02:53:39.6397798Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:53:39.638923] 2022-11-23T02:53:43.0840139Z 2022-11-23T02:53:43.0841251Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_tp_integration 2022-11-23T02:53:43.0843731Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_bueau2fv) 2022-11-23T02:53:43.0846124Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration 2022-11-23T02:53:43.0846912Z 2022-11-23T02:53:43.0847167Z Running tests... 2022-11-23T02:53:43.0848301Z ---------------------------------------------------------------------- 2022-11-23T02:53:43.0849452Z test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration) 2022-11-23T02:53:43.0851479Z Tests checkpointing for TP + FSDP integration. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.487s) 2022-11-23T02:53:43.0853185Z test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration) 2022-11-23T02:53:43.0855974Z Tests training for TP + FSDP integration by comparing an FSDP-only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.007s) 2022-11-23T02:53:43.0857756Z test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration) 2022-11-23T02:53:43.0860054Z Tests training for TP + FSDP integration by comparing an FSDP-only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:43.0861809Z test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration) 2022-11-23T02:53:43.0864079Z Tests training for TP + FSDP integration by comparing an FSDP-only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:43.0865781Z test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration) 2022-11-23T02:53:43.0868030Z Tests training for TP + FSDP integration by comparing an FSDP-only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:43.0869030Z 2022-11-23T02:53:43.0869771Z ---------------------------------------------------------------------- 2022-11-23T02:53:43.0870643Z Ran 5 tests in 0.501s 2022-11-23T02:53:43.0871049Z 2022-11-23T02:53:43.0871295Z OK (skipped=5) 2022-11-23T02:53:43.0871680Z 2022-11-23T02:53:43.0871977Z Generating XML reports... 2022-11-23T02:53:43.0873746Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221123025341.xml 2022-11-23T02:53:43.0874783Z 2022-11-23T02:53:43.0875553Z ##[endgroup] 2022-11-23T02:53:43.0877476Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_bueau2fv) 2022-11-23T02:53:43.0878476Z 2022-11-23T02:53:43.0879271Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-11-23 02:53:43.084646] 2022-11-23T02:53:43.0881402Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:53:43.085503] 2022-11-23T02:53:46.7485163Z 2022-11-23T02:53:46.7486456Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict 2022-11-23T02:53:46.7489576Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_state_dict_wuxupmqn) 2022-11-23T02:53:46.7492609Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict 2022-11-23T02:53:46.7493395Z 2022-11-23T02:53:46.7493656Z Running tests... 2022-11-23T02:53:46.7494831Z ---------------------------------------------------------------------- 2022-11-23T02:53:46.7496540Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7499078Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.502s) 2022-11-23T02:53:46.7501253Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7503782Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7505997Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7508596Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7510564Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7513051Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7515073Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7517775Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7519743Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7522211Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7524181Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7526638Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7528603Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7531054Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7532997Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7535657Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7537638Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7540105Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7542066Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7544536Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7546634Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7549085Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7551041Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7553473Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7555457Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7558004Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7559947Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7562370Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7564330Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7566783Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7568777Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7571256Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7573234Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7575834Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7577801Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7580283Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7582250Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7584706Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7586821Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7589314Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7591274Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7593738Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7595745Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7598318Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7599999Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7600977Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7601773Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7602752Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7603551Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7604529Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7605319Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7606368Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7607147Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7608111Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7608897Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7609848Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7610685Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7611656Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7612444Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7613418Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7614200Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7615162Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7615929Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7616891Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7617668Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7618636Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7619403Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7620363Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7621132Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7622147Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7622918Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7623876Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7624616Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7625572Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7626382Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7627341Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7628099Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7629055Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7629825Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7630780Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7631543Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7632508Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7633282Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7634241Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7635008Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7635957Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7636763Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7637779Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7638562Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7639500Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7640282Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7641247Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7642064Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7643023Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7644033Z test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7645085Z test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7646134Z test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7646902Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7647948Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7648705Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7649657Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7650402Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7651328Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7652064Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7652990Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7653719Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7654700Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7655433Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7656341Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7657094Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7658037Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7658780Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7659771Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7660520Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7661440Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7662174Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7663103Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7663843Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7664779Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7665511Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7666444Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7667140Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7668062Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7668779Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7669701Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7670489Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7671399Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7672127Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7673041Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7673782Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7674724Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7675467Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7676504Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7677233Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7678133Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7678861Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7679781Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7680526Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7681474Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7682213Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7683147Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7683887Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7684820Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7685554Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7686476Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7687270Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7688208Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7688940Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7689832Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7690561Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7691657Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7692438Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7693371Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7694093Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7695008Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7695728Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7696645Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7697359Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7698276Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7698989Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7699903Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7719859Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7720953Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7721675Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7722569Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7723254Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7724345Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7725056Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7725945Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7726638Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7727534Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:53:46.7728246Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7729194Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:53:46.7729889Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7730780Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:53:46.7731469Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7732352Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:53:46.7732972Z test_state_dict_rank0_offload_save_load_flow_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7733809Z Tests saving a model checkpoint only on rank 0 and loading it only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:53:46.7734434Z test_state_dict_rank0_offload_save_load_flow_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7735267Z Tests saving a model checkpoint only on rank 0 and loading it only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7736285Z test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7737362Z test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7738406Z test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7739460Z test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7740621Z test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7741779Z test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7742740Z test_state_dict_type (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7743836Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7745042Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7746287Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7747473Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7748643Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7749816Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7750985Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7752157Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7752919Z test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7753816Z Tests saving and loading a state dict for a model manually wrapped with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7754527Z test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7755425Z Tests saving and loading a state dict for a model manually wrapped with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7756162Z test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7757046Z Tests saving and loading a state dict for a model manually wrapped with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7757719Z test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T02:53:46.7758653Z Tests saving and loading a state dict for a model manually wrapped with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:53:46.7759709Z test_state_dict_with_shared_parameters_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7760806Z test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7761862Z test_state_dict_with_shared_parameters_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7762857Z test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:53:46.7763253Z 2022-11-23T02:53:46.7763561Z ---------------------------------------------------------------------- 2022-11-23T02:53:46.7763917Z Ran 116 tests in 0.698s 2022-11-23T02:53:46.7764089Z 2022-11-23T02:53:46.7764249Z OK (skipped=116) 2022-11-23T02:53:46.7764415Z 2022-11-23T02:53:46.7764543Z Generating XML reports... 2022-11-23T02:53:46.7765228Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123025344.xml 2022-11-23T02:53:46.7765607Z 2022-11-23T02:53:46.7766075Z ##[endgroup] 2022-11-23T02:53:46.7766744Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_state_dict_wuxupmqn) 2022-11-23T02:53:46.7767128Z 2022-11-23T02:53:46.7767439Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2022-11-23 02:53:46.749901] 2022-11-23T02:53:46.7768278Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:53:46.750757] 2022-11-23T02:58:53.0160671Z 2022-11-23T02:58:53.0161956Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_pure_fp16 2022-11-23T02:58:53.0165566Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_ckya24eo) 2022-11-23T02:58:53.0169213Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16 2022-11-23T02:58:53.0169709Z 2022-11-23T02:58:53.0169938Z Running tests... 2022-11-23T02:58:53.0170952Z ---------------------------------------------------------------------- 2022-11-23T02:58:53.0171598Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0172556Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5572 2022-11-23T02:58:53.0173539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5573 2022-11-23T02:58:53.0174518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0175122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0175896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0176512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0177150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0178211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0179753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0180676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0181300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0181891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0182778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0183998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0184712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0185318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0185768Z dist init r=0, world=2 2022-11-23T02:58:53.0187534Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0188635Z warnings.warn( 2022-11-23T02:58:53.0188953Z dist init r=1, world=2 2022-11-23T02:58:53.0190576Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0191598Z warnings.warn( 2022-11-23T02:58:53.0191891Z ok (6.229s) 2022-11-23T02:58:53.0192323Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0193376Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 50 2022-11-23T02:58:53.0194236Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5709 2022-11-23T02:58:53.0195180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5710 2022-11-23T02:58:53.0196465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0197320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0198486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0199421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0200306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0201715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0202520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0203689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0204506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0205346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0206584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0208197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0209149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0210050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0210677Z dist init r=0, world=2 2022-11-23T02:58:53.0211144Z dist init r=1, world=2 2022-11-23T02:58:53.0213522Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0214972Z warnings.warn( 2022-11-23T02:58:53.0217425Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0218904Z warnings.warn( 2022-11-23T02:58:53.0219304Z ok (5.734s) 2022-11-23T02:58:53.0219925Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0221303Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 49 2022-11-23T02:58:53.0222368Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5846 2022-11-23T02:58:53.0223261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5847 2022-11-23T02:58:53.0224393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0225207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0226293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0227133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0228013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0229249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0230098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0231209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0232066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0232943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0234164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0235464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0236533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0237415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0238097Z dist init r=1, world=2 2022-11-23T02:58:53.0240515Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0242046Z warnings.warn( 2022-11-23T02:58:53.0242457Z dist init r=0, world=2 2022-11-23T02:58:53.0244794Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0246275Z warnings.warn( 2022-11-23T02:58:53.0246739Z ok (6.436s) 2022-11-23T02:58:53.0247348Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0248761Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 48 2022-11-23T02:58:53.0249979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5983 2022-11-23T02:58:53.0250966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5984 2022-11-23T02:58:53.0251973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0252547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0253310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0253901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0254489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0255318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0255881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0256641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0257231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0257811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0258668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0259579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0260254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0260858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0261307Z dist init r=0, world=2 2022-11-23T02:58:53.0262916Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0263916Z warnings.warn( 2022-11-23T02:58:53.0264222Z dist init r=1, world=2 2022-11-23T02:58:53.0265819Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0266878Z warnings.warn( 2022-11-23T02:58:53.0267149Z ok (5.847s) 2022-11-23T02:58:53.0267581Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0268520Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 47 2022-11-23T02:58:53.0269222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6120 2022-11-23T02:58:53.0269791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6121 2022-11-23T02:58:53.0270593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0271168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0272020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0272629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0273208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0274037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0274604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0275363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0275961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0276677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0277542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0278463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0279034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0279425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0279716Z dist init r=1, world=2 2022-11-23T02:58:53.0280742Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0281404Z warnings.warn( 2022-11-23T02:58:53.0281609Z dist init r=0, world=2 2022-11-23T02:58:53.0282623Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0283262Z warnings.warn( 2022-11-23T02:58:53.0283441Z ok (6.239s) 2022-11-23T02:58:53.0283724Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0284389Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 46 2022-11-23T02:58:53.0284845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6257 2022-11-23T02:58:53.0285221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6258 2022-11-23T02:58:53.0285741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0286108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0286695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0287072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0287543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0288026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0288351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0288829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0289176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0289502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0290009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0290533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0290926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0291282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0291542Z dist init r=0, world=2 2022-11-23T02:58:53.0292458Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0293040Z warnings.warn( 2022-11-23T02:58:53.0293225Z dist init r=1, world=2 2022-11-23T02:58:53.0294140Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0294717Z warnings.warn( 2022-11-23T02:58:53.0294880Z ok (5.750s) 2022-11-23T02:58:53.0295134Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0295671Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 45 2022-11-23T02:58:53.0296078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6394 2022-11-23T02:58:53.0296416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6395 2022-11-23T02:58:53.0296880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0297253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0297684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0298033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0298373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0298854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0299184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0299623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0299967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0300297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0300805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0301374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0301762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0302112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0303047Z dist init r=1, world=2 2022-11-23T02:58:53.0303232Z dist init r=0, world=2 2022-11-23T02:58:53.0304145Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0304731Z warnings.warn( 2022-11-23T02:58:53.0305627Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0306218Z warnings.warn( 2022-11-23T02:58:53.0306378Z ok (5.740s) 2022-11-23T02:58:53.0306630Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0307170Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 44 2022-11-23T02:58:53.0307581Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6531 2022-11-23T02:58:53.0307916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6532 2022-11-23T02:58:53.0308380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0308716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0309146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0309495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0309835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0310315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0310690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0311131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0311479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0311803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0312308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0312836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0313226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0313578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0313840Z dist init r=1, world=2 2022-11-23T02:58:53.0314797Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0315380Z warnings.warn( 2022-11-23T02:58:53.0315561Z dist init r=0, world=2 2022-11-23T02:58:53.0316717Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0317353Z warnings.warn( 2022-11-23T02:58:53.0317543Z ok (5.837s) 2022-11-23T02:58:53.0317827Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0318434Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 43 2022-11-23T02:58:53.0318887Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6668 2022-11-23T02:58:53.0319258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6669 2022-11-23T02:58:53.0319772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0320130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0320620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0321005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0321389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0321921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0322288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0322776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0323149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0323523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0324139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0324728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0325166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0325556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0325847Z dist init r=1, world=2 2022-11-23T02:58:53.0326976Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0327817Z warnings.warn( 2022-11-23T02:58:53.0327987Z dist init r=0, world=2 2022-11-23T02:58:53.0328926Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0329492Z warnings.warn( 2022-11-23T02:58:53.0329666Z ok (5.838s) 2022-11-23T02:58:53.0329920Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0330460Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 42 2022-11-23T02:58:53.0330868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6805 2022-11-23T02:58:53.0331203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6806 2022-11-23T02:58:53.0331666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0331992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0332433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0332782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0333121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0333601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0333930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0334375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0334712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0335050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0335556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0336082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0336471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0336824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0337083Z dist init r=0, world=2 2022-11-23T02:58:53.0338035Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0338608Z warnings.warn( 2022-11-23T02:58:53.0338779Z dist init r=1, world=2 2022-11-23T02:58:53.0339683Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0340252Z warnings.warn( 2022-11-23T02:58:53.0340422Z ok (5.639s) 2022-11-23T02:58:53.0340677Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0341250Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 41 2022-11-23T02:58:53.0341657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6942 2022-11-23T02:58:53.0341991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6943 2022-11-23T02:58:53.0342444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0342779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0343221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0343568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0343906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0344384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0344712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0345150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0345485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0345822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0346323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0346858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0347245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0347600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0347862Z dist init r=0, world=2 2022-11-23T02:58:53.0348762Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0349355Z warnings.warn( 2022-11-23T02:58:53.0349526Z dist init r=1, world=2 2022-11-23T02:58:53.0350485Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0351055Z warnings.warn( 2022-11-23T02:58:53.0351227Z ok (6.137s) 2022-11-23T02:58:53.0351482Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0352019Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 40 2022-11-23T02:58:53.0352421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7079 2022-11-23T02:58:53.0352759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7080 2022-11-23T02:58:53.0353211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0353546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0354019Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0354371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0354710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0355189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0355525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0355956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0356560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0357007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0357577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0358169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0358603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0358999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0359292Z dist init r=1, world=2 2022-11-23T02:58:53.0360301Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0360955Z warnings.warn( 2022-11-23T02:58:53.0361162Z dist init r=0, world=2 2022-11-23T02:58:53.0362177Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0362827Z warnings.warn( 2022-11-23T02:58:53.0363077Z ok (5.735s) 2022-11-23T02:58:53.0363366Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0363974Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 39 2022-11-23T02:58:53.0364435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7216 2022-11-23T02:58:53.0364796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7217 2022-11-23T02:58:53.0365319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0365692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0366186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0366581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0367066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0367791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0368127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0368597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0368945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0369287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0369797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0370330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0370725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0371078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0371330Z dist init r=0, world=2 2022-11-23T02:58:53.0372251Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0372837Z warnings.warn( 2022-11-23T02:58:53.0373022Z dist init r=1, world=2 2022-11-23T02:58:53.0373930Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0374512Z warnings.warn( 2022-11-23T02:58:53.0374683Z ok (5.937s) 2022-11-23T02:58:53.0374937Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0375477Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 38 2022-11-23T02:58:53.0375890Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7353 2022-11-23T02:58:53.0376213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7354 2022-11-23T02:58:53.0376739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0377078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0377521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0377874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0378214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0378693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0379013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0379454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0379802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0380137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0380641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0381206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0381599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0381950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0382198Z dist init r=1, world=2 2022-11-23T02:58:53.0383114Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0383694Z warnings.warn( 2022-11-23T02:58:53.0383877Z dist init r=0, world=2 2022-11-23T02:58:53.0384787Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0385363Z warnings.warn( 2022-11-23T02:58:53.0385533Z ok (5.838s) 2022-11-23T02:58:53.0385790Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0386330Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 37 2022-11-23T02:58:53.0386727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7490 2022-11-23T02:58:53.0387066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7491 2022-11-23T02:58:53.0387527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0387864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0388304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0388653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0388994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0389513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0389833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0390274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0390617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0390953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0391455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0391987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0392376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0392718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0392981Z dist init r=1, world=2 2022-11-23T02:58:53.0393931Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0394518Z warnings.warn( 2022-11-23T02:58:53.0394702Z dist init r=0, world=2 2022-11-23T02:58:53.0395613Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0396432Z warnings.warn( 2022-11-23T02:58:53.0396628Z ok (6.040s) 2022-11-23T02:58:53.0396918Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0397526Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 36 2022-11-23T02:58:53.0397968Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7627 2022-11-23T02:58:53.0398339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7628 2022-11-23T02:58:53.0398851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0399229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0413767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0414191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0414548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0415109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0415453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0415913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0416268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0416597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0417284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0417817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0418212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0418564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0418828Z dist init r=1, world=2 2022-11-23T02:58:53.0419753Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0420349Z warnings.warn( 2022-11-23T02:58:53.0420536Z dist init r=0, world=2 2022-11-23T02:58:53.0421472Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0422053Z warnings.warn( 2022-11-23T02:58:53.0422226Z ok (6.240s) 2022-11-23T02:58:53.0422487Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0423032Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 35 2022-11-23T02:58:53.0423446Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7764 2022-11-23T02:58:53.0423783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7765 2022-11-23T02:58:53.0424250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0424578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0425023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0425374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0425718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0426199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0426536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0426976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0427325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0427652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0428156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0428691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0429080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0429430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0429727Z dist init r=1, world=2 2022-11-23T02:58:53.0430641Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0431227Z warnings.warn( 2022-11-23T02:58:53.0431398Z dist init r=0, world=2 2022-11-23T02:58:53.0432301Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0432876Z warnings.warn( 2022-11-23T02:58:53.0433048Z ok (5.738s) 2022-11-23T02:58:53.0433303Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0433875Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 34 2022-11-23T02:58:53.0434287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7901 2022-11-23T02:58:53.0434622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7902 2022-11-23T02:58:53.0435085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0435409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0435853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0436450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0436833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0437373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0437741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0438234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0438613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0438987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0439549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0440145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0440585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0440979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0441271Z dist init r=1, world=2 2022-11-23T02:58:53.0442287Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0442995Z warnings.warn( 2022-11-23T02:58:53.0443184Z dist init r=0, world=2 2022-11-23T02:58:53.0444210Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0444850Z warnings.warn( 2022-11-23T02:58:53.0445043Z ok (6.037s) 2022-11-23T02:58:53.0445332Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0445938Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 33 2022-11-23T02:58:53.0446394Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8038 2022-11-23T02:58:53.0446769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8039 2022-11-23T02:58:53.0447476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0447854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0448305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0448653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0448994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0449476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0449805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0450246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0450581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0450917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0451425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0451947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0452340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0452694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0452956Z dist init r=1, world=2 2022-11-23T02:58:53.0453871Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0454459Z warnings.warn( 2022-11-23T02:58:53.0454630Z dist init r=0, world=2 2022-11-23T02:58:53.0455533Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0456127Z warnings.warn( 2022-11-23T02:58:53.0456304Z ok (5.938s) 2022-11-23T02:58:53.0456559Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0457103Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 32 2022-11-23T02:58:53.0457508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8175 2022-11-23T02:58:53.0457840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8176 2022-11-23T02:58:53.0458293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0458625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0459066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0459416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0459755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0460241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0460600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0461032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0461380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0461717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0462219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0462747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0463139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0463488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0463751Z dist init r=1, world=2 2022-11-23T02:58:53.0464648Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0465234Z warnings.warn( 2022-11-23T02:58:53.0465416Z dist init r=0, world=2 2022-11-23T02:58:53.0466325Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0466950Z warnings.warn( 2022-11-23T02:58:53.0467122Z ok (5.939s) 2022-11-23T02:58:53.0467378Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0467919Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 31 2022-11-23T02:58:53.0468328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8312 2022-11-23T02:58:53.0468651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8313 2022-11-23T02:58:53.0469148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0469484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0469927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0470278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0470618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0471101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0471431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0471856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0472203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0472541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0473074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0473606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0473994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0474345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0474595Z dist init r=0, world=2 2022-11-23T02:58:53.0475509Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0476159Z warnings.warn( 2022-11-23T02:58:53.0476512Z dist init r=1, world=2 2022-11-23T02:58:53.0477526Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0478176Z warnings.warn( 2022-11-23T02:58:53.0478365Z ok (5.739s) 2022-11-23T02:58:53.0478653Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0479259Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 30 2022-11-23T02:58:53.0479716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8449 2022-11-23T02:58:53.0480080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8450 2022-11-23T02:58:53.0480590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0480963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0481457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0481848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0482227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0482819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0483181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0483676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0484067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0484466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0485035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0485629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0486065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0486455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0486737Z dist init r=1, world=2 2022-11-23T02:58:53.0486939Z dist init r=0, world=2 2022-11-23T02:58:53.0488171Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0488759Z warnings.warn( 2022-11-23T02:58:53.0489665Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0490238Z warnings.warn( 2022-11-23T02:58:53.0490410Z ok (5.739s) 2022-11-23T02:58:53.0490673Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0491226Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 29 2022-11-23T02:58:53.0491647Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8586 2022-11-23T02:58:53.0491973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8587 2022-11-23T02:58:53.0492452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0492809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0493269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0493642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0493991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0494488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0494814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0495257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0495606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0495972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0496479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0497009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0497399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0497747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0498001Z dist init r=1, world=2 2022-11-23T02:58:53.0498915Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0499496Z warnings.warn( 2022-11-23T02:58:53.0499684Z dist init r=0, world=2 2022-11-23T02:58:53.0500615Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0501187Z warnings.warn( 2022-11-23T02:58:53.0501359Z ok (5.938s) 2022-11-23T02:58:53.0501614Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0502157Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 28 2022-11-23T02:58:53.0502558Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8723 2022-11-23T02:58:53.0502891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8724 2022-11-23T02:58:53.0503349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0503685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0504128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0504477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0504815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0505286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0505621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0506062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0506409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0506745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0507245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0507772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0508160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0508532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0508793Z dist init r=1, world=2 2022-11-23T02:58:53.0509713Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0510298Z warnings.warn( 2022-11-23T02:58:53.0510476Z dist init r=0, world=2 2022-11-23T02:58:53.0511374Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0511954Z warnings.warn( 2022-11-23T02:58:53.0512126Z ok (5.738s) 2022-11-23T02:58:53.0512404Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0512926Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 27 2022-11-23T02:58:53.0513326Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8860 2022-11-23T02:58:53.0513655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8861 2022-11-23T02:58:53.0514114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0514441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0514879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0515224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0515564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0516031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0516638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0517127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0517505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0517874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0518438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0519023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0519450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0519826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0520100Z dist init r=1, world=2 2022-11-23T02:58:53.0520293Z dist init r=0, world=2 2022-11-23T02:58:53.0521367Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0522065Z warnings.warn( 2022-11-23T02:58:53.0523058Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0523703Z warnings.warn( 2022-11-23T02:58:53.0523888Z ok (5.637s) 2022-11-23T02:58:53.0524166Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0524756Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 26 2022-11-23T02:58:53.0525204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8997 2022-11-23T02:58:53.0525570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8998 2022-11-23T02:58:53.0526122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0526489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0527075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0527551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0527872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0528346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0528671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0529099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0529438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0529772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0530273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0530789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0531159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0531501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0531748Z dist init r=1, world=2 2022-11-23T02:58:53.0532652Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0533229Z warnings.warn( 2022-11-23T02:58:53.0533400Z dist init r=0, world=2 2022-11-23T02:58:53.0534295Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0534916Z warnings.warn( 2022-11-23T02:58:53.0535077Z ok (5.637s) 2022-11-23T02:58:53.0535319Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0535853Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 25 2022-11-23T02:58:53.0536249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9134 2022-11-23T02:58:53.0536575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9135 2022-11-23T02:58:53.0537027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0537352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0537781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0538122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0538450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0538949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0539266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0539700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0540036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0540372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0540865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0541386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0541756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0542096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0542350Z dist init r=1, world=2 2022-11-23T02:58:53.0543247Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0543821Z warnings.warn( 2022-11-23T02:58:53.0543995Z dist init r=0, world=2 2022-11-23T02:58:53.0544901Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0545465Z warnings.warn( 2022-11-23T02:58:53.0545626Z ok (5.738s) 2022-11-23T02:58:53.0545864Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0546392Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 24 2022-11-23T02:58:53.0546787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9271 2022-11-23T02:58:53.0547147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9272 2022-11-23T02:58:53.0547601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0547922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0548358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0548697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0549018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0549483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0549802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0550233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0550572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0550905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0551435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0551949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0552324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0552664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0552913Z dist init r=1, world=2 2022-11-23T02:58:53.0553816Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0554388Z warnings.warn( 2022-11-23T02:58:53.0554559Z dist init r=0, world=2 2022-11-23T02:58:53.0555467Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0556027Z warnings.warn( 2022-11-23T02:58:53.0556245Z ok (5.837s) 2022-11-23T02:58:53.0556671Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0557247Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 23 2022-11-23T02:58:53.0557679Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9408 2022-11-23T02:58:53.0558026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9409 2022-11-23T02:58:53.0558517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0558872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0559341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0559702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0560099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0560610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0560957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0561422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0561787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0562146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0562685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0563239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0563650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0564026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0564301Z dist init r=0, world=2 2022-11-23T02:58:53.0565324Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0565951Z warnings.warn( 2022-11-23T02:58:53.0566139Z dist init r=1, world=2 2022-11-23T02:58:53.0567125Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0567738Z warnings.warn( 2022-11-23T02:58:53.0567908Z ok (5.939s) 2022-11-23T02:58:53.0568176Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0568751Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 22 2022-11-23T02:58:53.0569179Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9545 2022-11-23T02:58:53.0569536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9546 2022-11-23T02:58:53.0570031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0570386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0570856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0571218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0571576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0572087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0572439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0572905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0573303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0573773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0574375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0574904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0575285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0575625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0575874Z dist init r=0, world=2 2022-11-23T02:58:53.0576044Z dist init r=1, world=2 2022-11-23T02:58:53.0576948Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0577522Z warnings.warn( 2022-11-23T02:58:53.0578446Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0579017Z warnings.warn( 2022-11-23T02:58:53.0579171Z ok (5.739s) 2022-11-23T02:58:53.0579415Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0579947Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 21 2022-11-23T02:58:53.0580344Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9682 2022-11-23T02:58:53.0580670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9683 2022-11-23T02:58:53.0581120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0581441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0581870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0582210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0582540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0583012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0583335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0583771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0584106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0584439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0584931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0585454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0585836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0586243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0586495Z dist init r=1, world=2 2022-11-23T02:58:53.0587406Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0587978Z warnings.warn( 2022-11-23T02:58:53.0588151Z dist init r=0, world=2 2022-11-23T02:58:53.0589044Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0589610Z warnings.warn( 2022-11-23T02:58:53.0589767Z ok (6.035s) 2022-11-23T02:58:53.0590040Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0590567Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 20 2022-11-23T02:58:53.0590970Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9819 2022-11-23T02:58:53.0591297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9820 2022-11-23T02:58:53.0591749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0592080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0592508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0592847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0593179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0593647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0593971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0594403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0594740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0595061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0595563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0596132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0596703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0597075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0597350Z dist init r=0, world=2 2022-11-23T02:58:53.0598335Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0598998Z warnings.warn( 2022-11-23T02:58:53.0599188Z dist init r=1, world=2 2022-11-23T02:58:53.0600164Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0600781Z warnings.warn( 2022-11-23T02:58:53.0600957Z ok (5.737s) 2022-11-23T02:58:53.0601222Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0601794Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 19 2022-11-23T02:58:53.0602229Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9956 2022-11-23T02:58:53.0602585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9957 2022-11-23T02:58:53.0603110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0603458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0603932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0604302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0604661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0605166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0605516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0605985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0606356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0606705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0607364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0607887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0608268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0608610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0608867Z dist init r=1, world=2 2022-11-23T02:58:53.0609768Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0610344Z warnings.warn( 2022-11-23T02:58:53.0610508Z dist init r=0, world=2 2022-11-23T02:58:53.0611404Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0612004Z warnings.warn( 2022-11-23T02:58:53.0612172Z ok (5.936s) 2022-11-23T02:58:53.0612421Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0612958Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 18 2022-11-23T02:58:53.0613364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10093 2022-11-23T02:58:53.0613695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10094 2022-11-23T02:58:53.0614150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0614469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0614903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0615245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0615572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0616067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0616390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0616816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0617147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0617475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0617968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0618490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0618876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0619215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0619462Z dist init r=0, world=2 2022-11-23T02:58:53.0620363Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0620937Z warnings.warn( 2022-11-23T02:58:53.0621100Z dist init r=1, world=2 2022-11-23T02:58:53.0621996Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0622555Z warnings.warn( 2022-11-23T02:58:53.0622725Z ok (5.738s) 2022-11-23T02:58:53.0622970Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0623499Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 17 2022-11-23T02:58:53.0623931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10230 2022-11-23T02:58:53.0624260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10231 2022-11-23T02:58:53.0624711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0625039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0625472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0625809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0626140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0626610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0626933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0627361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0627693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0628047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0628549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0629069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0629455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0629797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0630049Z dist init r=0, world=2 2022-11-23T02:58:53.0630951Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0631522Z warnings.warn( 2022-11-23T02:58:53.0631686Z dist init r=1, world=2 2022-11-23T02:58:53.0632596Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0633168Z warnings.warn( 2022-11-23T02:58:53.0633335Z ok (6.039s) 2022-11-23T02:58:53.0633585Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0634118Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 16 2022-11-23T02:58:53.0634521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10367 2022-11-23T02:58:53.0634850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10368 2022-11-23T02:58:53.0635301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0635627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0636102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0636663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0637019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0637538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0637886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0638346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0638714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0639075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0639619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0640189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0640603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0641015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0641286Z dist init r=0, world=2 2022-11-23T02:58:53.0642269Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0642898Z warnings.warn( 2022-11-23T02:58:53.0643076Z dist init r=1, world=2 2022-11-23T02:58:53.0644057Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0644673Z warnings.warn( 2022-11-23T02:58:53.0644849Z ok (5.944s) 2022-11-23T02:58:53.0645115Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0645692Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 15 2022-11-23T02:58:53.0646130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10504 2022-11-23T02:58:53.0646489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10505 2022-11-23T02:58:53.0646976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0647331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0647911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0648270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0648618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0649115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0649458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0650003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0650382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0650715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0651218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0651741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0652125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0652466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0652713Z dist init r=1, world=2 2022-11-23T02:58:53.0653620Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0654227Z warnings.warn( 2022-11-23T02:58:53.0654403Z dist init r=0, world=2 2022-11-23T02:58:53.0655304Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0655868Z warnings.warn( 2022-11-23T02:58:53.0656032Z ok (5.737s) 2022-11-23T02:58:53.0656278Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0656807Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 14 2022-11-23T02:58:53.0657213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10641 2022-11-23T02:58:53.0657535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10642 2022-11-23T02:58:53.0657989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0658317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0658753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0659092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0659429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0659896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0660222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0660653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0660991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0661324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0661821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0662342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0662757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0663100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0663345Z dist init r=1, world=2 2022-11-23T02:58:53.0664252Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0664826Z warnings.warn( 2022-11-23T02:58:53.0665003Z dist init r=0, world=2 2022-11-23T02:58:53.0665901Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0666499Z warnings.warn( 2022-11-23T02:58:53.0666667Z ok (6.037s) 2022-11-23T02:58:53.0666914Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0667447Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 13 2022-11-23T02:58:53.0667846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10778 2022-11-23T02:58:53.0668170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10779 2022-11-23T02:58:53.0668627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0668954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0669392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0669737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0670068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0670539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0670860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0671292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0671631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0671966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0672461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0672987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0673373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0673716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0673958Z dist init r=0, world=2 2022-11-23T02:58:53.0674878Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0675482Z warnings.warn( 2022-11-23T02:58:53.0675658Z dist init r=1, world=2 2022-11-23T02:58:53.0676810Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0677430Z warnings.warn( 2022-11-23T02:58:53.0677602Z ok (5.838s) 2022-11-23T02:58:53.0677867Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0678443Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 12 2022-11-23T02:58:53.0678873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10915 2022-11-23T02:58:53.0679289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10916 2022-11-23T02:58:53.0679785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0680135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0680604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0680968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0681326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0681840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0682188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0682662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0683028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0683385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0683925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0684501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0684915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0685287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0685558Z dist init r=0, world=2 2022-11-23T02:58:53.0686540Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0687165Z warnings.warn( 2022-11-23T02:58:53.0687468Z dist init r=1, world=2 2022-11-23T02:58:53.0688370Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0688966Z warnings.warn( 2022-11-23T02:58:53.0689130Z ok (6.237s) 2022-11-23T02:58:53.0689378Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0689913Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 11 2022-11-23T02:58:53.0690302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11052 2022-11-23T02:58:53.0690633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11053 2022-11-23T02:58:53.0691092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0691426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0691862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0692207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0692566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0693038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0693363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0693791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0694134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0694461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0694962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0695484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0695865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0696202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0696450Z dist init r=1, world=2 2022-11-23T02:58:53.0697357Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0697923Z warnings.warn( 2022-11-23T02:58:53.0697996Z dist init r=0, world=2 2022-11-23T02:58:53.0698799Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0698869Z warnings.warn( 2022-11-23T02:58:53.0698930Z ok (6.440s) 2022-11-23T02:58:53.0699087Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0699448Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 10 2022-11-23T02:58:53.0699646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11189 2022-11-23T02:58:53.0699805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11190 2022-11-23T02:58:53.0700095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0700222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0700516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0700654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0700832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0701115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0701245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0701536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0701661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0701878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0702193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0702501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0702668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0702833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0702912Z dist init r=1, world=2 2022-11-23T02:58:53.0703714Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0703791Z warnings.warn( 2022-11-23T02:58:53.0703865Z dist init r=0, world=2 2022-11-23T02:58:53.0704662Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0704738Z warnings.warn( 2022-11-23T02:58:53.0704806Z ok (6.541s) 2022-11-23T02:58:53.0704964Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0705337Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 9 2022-11-23T02:58:53.0711610Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11326 2022-11-23T02:58:53.0711810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11327 2022-11-23T02:58:53.0712136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0712261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0712556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0712790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0712968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0713258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0713380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0713667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0713801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0713978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0714279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0714588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0714749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0714952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0715024Z dist init r=1, world=2 2022-11-23T02:58:53.0715836Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0715910Z warnings.warn( 2022-11-23T02:58:53.0715979Z dist init r=0, world=2 2022-11-23T02:58:53.0717060Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0717135Z warnings.warn( 2022-11-23T02:58:53.0717201Z ok (5.935s) 2022-11-23T02:58:53.0717368Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0717754Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 8 2022-11-23T02:58:53.0717925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11463 2022-11-23T02:58:53.0718093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11464 2022-11-23T02:58:53.0718398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0718531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0718848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0718991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0719181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0719487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0719617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0719973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0720117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0720309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0720641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0720969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0721139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0721312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0721389Z dist init r=1, world=2 2022-11-23T02:58:53.0722287Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0722366Z warnings.warn( 2022-11-23T02:58:53.0722439Z dist init r=0, world=2 2022-11-23T02:58:53.0723297Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0723376Z warnings.warn( 2022-11-23T02:58:53.0723443Z ok (6.640s) 2022-11-23T02:58:53.0723612Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0724000Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 7 2022-11-23T02:58:53.0724169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11600 2022-11-23T02:58:53.0724333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11601 2022-11-23T02:58:53.0724636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0724768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0725081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0725227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0725414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0725715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0725848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0726160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0726301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0726489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0726823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0727150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0727472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0727742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0727809Z dist init r=1, world=2 2022-11-23T02:58:53.0728613Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0728685Z warnings.warn( 2022-11-23T02:58:53.0728758Z dist init r=0, world=2 2022-11-23T02:58:53.0729584Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0729657Z warnings.warn( 2022-11-23T02:58:53.0729718Z ok (5.738s) 2022-11-23T02:58:53.0729872Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0730231Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 6 2022-11-23T02:58:53.0730388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11737 2022-11-23T02:58:53.0730542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11738 2022-11-23T02:58:53.0730826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0730946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0731240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0731371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0731544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0731820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0731941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0732225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0732358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0732533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0732840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0733138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0733299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0733459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0733531Z dist init r=1, world=2 2022-11-23T02:58:53.0734340Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0734441Z warnings.warn( 2022-11-23T02:58:53.0734506Z dist init r=0, world=2 2022-11-23T02:58:53.0735313Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0735383Z warnings.warn( 2022-11-23T02:58:53.0735437Z ok (6.237s) 2022-11-23T02:58:53.0735593Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0735944Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 5 2022-11-23T02:58:53.0736129Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11874 2022-11-23T02:58:53.0736283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11875 2022-11-23T02:58:53.0736563Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0736680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0736968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0737099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0737276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0737555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0737675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0737960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0738090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0738263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0738568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0738868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0739033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0739196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0739269Z dist init r=0, world=2 2022-11-23T02:58:53.0740072Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0740146Z warnings.warn( 2022-11-23T02:58:53.0740213Z dist init r=1, world=2 2022-11-23T02:58:53.0741013Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0741117Z warnings.warn( 2022-11-23T02:58:53.0741181Z ok (6.038s) 2022-11-23T02:58:53.0741328Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0741686Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 4 2022-11-23T02:58:53.0741841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12011 2022-11-23T02:58:53.0741996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12012 2022-11-23T02:58:53.0742275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0742399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0742687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0742849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0743023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0743297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0743413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0743698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0743828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0744004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0744309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0744610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0744770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0744924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0744995Z dist init r=0, world=2 2022-11-23T02:58:53.0745788Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0745862Z warnings.warn( 2022-11-23T02:58:53.0745932Z dist init r=1, world=2 2022-11-23T02:58:53.0746728Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0746798Z warnings.warn( 2022-11-23T02:58:53.0746861Z ok (6.139s) 2022-11-23T02:58:53.0747014Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0747419Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 3 2022-11-23T02:58:53.0747567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12148 2022-11-23T02:58:53.0747720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12149 2022-11-23T02:58:53.0747997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0748118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0748407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0748538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0748717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0748998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0749118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0749404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0749583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0749757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0750065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0750370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0750532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0750694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0750767Z dist init r=1, world=2 2022-11-23T02:58:53.0751560Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0751631Z warnings.warn( 2022-11-23T02:58:53.0751702Z dist init r=0, world=2 2022-11-23T02:58:53.0752500Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0752575Z warnings.warn( 2022-11-23T02:58:53.0752638Z ok (5.838s) 2022-11-23T02:58:53.0752793Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0753151Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 2 2022-11-23T02:58:53.0753307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12285 2022-11-23T02:58:53.0753453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12286 2022-11-23T02:58:53.0753737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0753856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0754177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0754308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0754486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0754762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0754883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0755169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0755300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0755475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0755788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0756144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0756510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0756687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0756765Z dist init r=0, world=2 2022-11-23T02:58:53.0757634Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0757714Z warnings.warn( 2022-11-23T02:58:53.0757789Z dist init r=1, world=2 2022-11-23T02:58:53.0758664Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0758740Z warnings.warn( 2022-11-23T02:58:53.0758806Z ok (5.936s) 2022-11-23T02:58:53.0758973Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T02:58:53.0759355Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 1 2022-11-23T02:58:53.0759527Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12422 2022-11-23T02:58:53.0759692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12423 2022-11-23T02:58:53.0759995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0760120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0760438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0760583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0760771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:58:53.0761072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:58:53.0761237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:58:53.0761551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:58:53.0761693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:58:53.0761883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:58:53.0762216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0762543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:58:53.0762715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:58:53.0762889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:58:53.0762971Z dist init r=1, world=2 2022-11-23T02:58:53.0763877Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0763957Z warnings.warn( 2022-11-23T02:58:53.0764032Z dist init r=0, world=2 2022-11-23T02:58:53.0764904Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T02:58:53.0764981Z warnings.warn( 2022-11-23T02:58:53.0765048Z ok (5.938s) 2022-11-23T02:58:53.0765217Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16) 2022-11-23T02:58:53.0765603Z Tests pure FP16 training, including when the parameter's dtype is ... test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 0 2022-11-23T02:58:53.0765915Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:58:53.0765924Z 2022-11-23T02:58:53.0766148Z ---------------------------------------------------------------------- 2022-11-23T02:58:53.0766229Z Ran 52 tests in 303.264s 2022-11-23T02:58:53.0766234Z 2022-11-23T02:58:53.0766307Z OK (skipped=1) 2022-11-23T02:58:53.0766312Z 2022-11-23T02:58:53.0766397Z Generating XML reports... 2022-11-23T02:58:53.0766747Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221123025348.xml 2022-11-23T02:58:53.0766755Z 2022-11-23T02:58:53.0767143Z ##[endgroup] 2022-11-23T02:58:53.0767635Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_ckya24eo) 2022-11-23T02:58:53.0767642Z 2022-11-23T02:58:53.0767857Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-11-23 02:58:53.020427] 2022-11-23T02:58:53.0768282Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:58:53.021294] 2022-11-23T02:58:56.5885832Z 2022-11-23T02:58:56.5887076Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_optim_state 2022-11-23T02:58:56.5889438Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_optim_state_fdsdajmt) 2022-11-23T02:58:56.5892268Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state 2022-11-23T02:58:56.5893039Z 2022-11-23T02:58:56.5893295Z Running tests... 2022-11-23T02:58:56.5894416Z ---------------------------------------------------------------------- 2022-11-23T02:58:56.5895585Z test_flatten_sharded_optim_state_dict_nested (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5897934Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.522s) 2022-11-23T02:58:56.5899616Z test_flatten_sharded_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5901719Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5903209Z test_full_optim_state_dict_keys (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5905082Z Tests that the parameter keys returned by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5906704Z test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5908948Z Tests that :meth:`full_optim_state_dict` raises an error when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5910340Z test_optim_input_warning (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5912339Z Tests that passing the ``optim_input`` argument into optimizer state ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5914255Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5916869Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5918800Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5921208Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5923096Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5925491Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5927388Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5929778Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5931705Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5934115Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5936007Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5938550Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5940456Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5942846Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5944708Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5947110Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5949141Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5951578Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5953473Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5955899Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5957959Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5960397Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5962290Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5964698Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5966584Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5969006Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5970887Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5973290Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5975175Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5977717Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5979629Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5982018Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5983786Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5986001Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5987765Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5989990Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5991857Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5994116Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5995830Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.5998110Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.5999524Z test_rekey_optim_state_dict_to_names (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6001486Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6003123Z test_save_load_without_0th_param_state_state_dict_type_StateDictType_FULL_STATE_DICT (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6005332Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:58:56.6007034Z test_save_load_without_0th_param_state_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6009270Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:58:56.6010839Z test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6012924Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6014714Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6017001Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6018778Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6021188Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6022961Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6025186Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6026956Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6029182Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6030964Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6033200Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6035104Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6037472Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6039361Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6040260Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6040945Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6041838Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6042453Z test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6043253Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6043870Z test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6044693Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6045408Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6046300Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6047009Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6047909Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6048621Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6049546Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6050256Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6051198Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6051912Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6052820Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6053525Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6054471Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6055179Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6056084Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6056787Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6057690Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6058299Z test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6059109Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6059797Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6060706Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6061421Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6062333Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6063072Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6063991Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6064712Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6065624Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6066250Z test_use_orig_params_error (__main__.TestFSDPOptimState) 2022-11-23T02:58:56.6067046Z Tests that the optimizer state checkpointing APIs raise an error ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:58:56.6067442Z 2022-11-23T02:58:56.6067753Z ---------------------------------------------------------------------- 2022-11-23T02:58:56.6068094Z Ran 53 tests in 0.564s 2022-11-23T02:58:56.6068261Z 2022-11-23T02:58:56.6068367Z OK (skipped=53) 2022-11-23T02:58:56.6068525Z 2022-11-23T02:58:56.6068654Z Generating XML reports... 2022-11-23T02:58:56.6069339Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221123025854.xml 2022-11-23T02:58:56.6069723Z 2022-11-23T02:58:56.6070102Z ##[endgroup] 2022-11-23T02:58:56.6070793Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_optim_state_fdsdajmt) 2022-11-23T02:58:56.6071182Z 2022-11-23T02:58:56.6071516Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-11-23 02:58:56.589612] 2022-11-23T02:58:56.6072409Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:58:56.590467] 2022-11-23T02:59:00.0936350Z 2022-11-23T02:59:00.0937634Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_forward 2022-11-23T02:59:00.0940050Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_6g7bnovl) 2022-11-23T02:59:00.0942386Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward 2022-11-23T02:59:00.0943397Z 2022-11-23T02:59:00.0943669Z Running tests... 2022-11-23T02:59:00.0944845Z ---------------------------------------------------------------------- 2022-11-23T02:59:00.0946842Z test_multi_forward (__main__.TestMultiForward) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.525s) 2022-11-23T02:59:00.0947800Z 2022-11-23T02:59:00.0948548Z ---------------------------------------------------------------------- 2022-11-23T02:59:00.0949386Z Ran 1 test in 0.525s 2022-11-23T02:59:00.0949788Z 2022-11-23T02:59:00.0950039Z OK (skipped=1) 2022-11-23T02:59:00.0950422Z 2022-11-23T02:59:00.0950717Z Generating XML reports... 2022-11-23T02:59:00.0952627Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123025858.xml 2022-11-23T02:59:00.0953587Z 2022-11-23T02:59:00.0954340Z ##[endgroup] 2022-11-23T02:59:00.0956276Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_6g7bnovl) 2022-11-23T02:59:00.0957302Z 2022-11-23T02:59:00.0958065Z Running distributed/fsdp/test_fsdp_misc ... [2022-11-23 02:59:00.094056] 2022-11-23T02:59:00.0960122Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:59:00.094918] 2022-11-23T02:59:03.6857589Z 2022-11-23T02:59:03.6859290Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_misc 2022-11-23T02:59:03.6861984Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_misc_3_zamj8f) 2022-11-23T02:59:03.6864217Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc 2022-11-23T02:59:03.6865040Z 2022-11-23T02:59:03.6865305Z Running tests... 2022-11-23T02:59:03.6866453Z ---------------------------------------------------------------------- 2022-11-23T02:59:03.6868123Z test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6870177Z Tests that passing ``sync_module_states=True`` raises an error for ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.521s) 2022-11-23T02:59:03.6871577Z test_device_id_auto_wrap (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6873519Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6874929Z test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6877677Z Tests that passing a CPU module to FSDP preserves that the wrapped ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6879730Z test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6882302Z Ensures that even if device_id is specified but we have ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6884345Z test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6887168Z Tests the FSDP ``device_id`` argument: ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:59:03.6888945Z test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6891327Z Tests the FSDP ``device_id`` argument: ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:59:03.6894808Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6899143Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6903470Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6907811Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6910687Z test_fsdp_namedtuple (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6913038Z test_fsdp_not_all_outputs_used_in_loss (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:59:03.6914474Z test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6916614Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6918100Z test_module_device_mismatches_device_id (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6920077Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6921501Z test_multi_device_not_supported (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6923476Z Tests that wrapping a multi-device module (i.e. with submodules on ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6925029Z test_no_params (__main__.TestFSDPMisc) 2022-11-23T02:59:03.6926913Z Test that device_id and cpu init work if module has no params ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:03.6927899Z 2022-11-23T02:59:03.6928662Z ---------------------------------------------------------------------- 2022-11-23T02:59:03.6929541Z Ran 16 tests in 0.540s 2022-11-23T02:59:03.6929958Z 2022-11-23T02:59:03.6930211Z OK (skipped=16) 2022-11-23T02:59:03.6930569Z 2022-11-23T02:59:03.6930868Z Generating XML reports... 2022-11-23T02:59:03.6932492Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123025901.xml 2022-11-23T02:59:03.6933401Z 2022-11-23T02:59:03.6934256Z ##[endgroup] 2022-11-23T02:59:03.6935901Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_misc_3_zamj8f) 2022-11-23T02:59:03.6936825Z 2022-11-23T02:59:03.6937599Z Running distributed/fsdp/test_fsdp_memory ... [2022-11-23 02:59:03.686839] 2022-11-23T02:59:03.6939851Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:59:03.690477] 2022-11-23T02:59:07.2166812Z 2022-11-23T02:59:07.2168419Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_memory 2022-11-23T02:59:07.2170979Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_memory_vmdsegng) 2022-11-23T02:59:07.2173208Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory 2022-11-23T02:59:07.2173965Z 2022-11-23T02:59:07.2174270Z Running tests... 2022-11-23T02:59:07.2175398Z ---------------------------------------------------------------------- 2022-11-23T02:59:07.2177461Z test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.531s) 2022-11-23T02:59:07.2179832Z test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:59:07.2180815Z 2022-11-23T02:59:07.2181550Z ---------------------------------------------------------------------- 2022-11-23T02:59:07.2182418Z Ran 2 tests in 0.533s 2022-11-23T02:59:07.2182825Z 2022-11-23T02:59:07.2183071Z OK (skipped=2) 2022-11-23T02:59:07.2183452Z 2022-11-23T02:59:07.2183743Z Generating XML reports... 2022-11-23T02:59:07.2185360Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221123025905.xml 2022-11-23T02:59:07.2186267Z 2022-11-23T02:59:07.2187187Z ##[endgroup] 2022-11-23T02:59:07.2188840Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_memory_vmdsegng) 2022-11-23T02:59:07.2189784Z 2022-11-23T02:59:07.2190611Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-11-23 02:59:07.217377] 2022-11-23T02:59:07.2192807Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:59:07.218257] 2022-11-23T02:59:10.7566828Z 2022-11-23T02:59:10.7568312Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_ignored_modules 2022-11-23T02:59:10.7570988Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_hoizw_sj) 2022-11-23T02:59:10.7573301Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules 2022-11-23T02:59:10.7574177Z 2022-11-23T02:59:10.7574917Z Running tests... 2022-11-23T02:59:10.7576088Z ---------------------------------------------------------------------- 2022-11-23T02:59:10.7577839Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules) 2022-11-23T02:59:10.7580003Z Tests ignoring different modules across ranks. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.525s) 2022-11-23T02:59:10.7581628Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules) 2022-11-23T02:59:10.7583686Z Tests ignoring different modules across ranks. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:10.7585216Z test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules) 2022-11-23T02:59:10.7587344Z Tests that passing an FSDP module as an ignored module or the ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:10.7588804Z test_ignored_modules_nested (__main__.TestFSDPIgnoredModules) 2022-11-23T02:59:10.7590821Z Tests that passing a module with nested FSDP modules does not ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:10.7592497Z test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules) 2022-11-23T02:59:10.7594546Z Tests that ignored modules' parameters are not flattened for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:10.7595572Z 2022-11-23T02:59:10.7596513Z ---------------------------------------------------------------------- 2022-11-23T02:59:10.7597386Z Ran 5 tests in 0.529s 2022-11-23T02:59:10.7597794Z 2022-11-23T02:59:10.7598045Z OK (skipped=5) 2022-11-23T02:59:10.7598433Z 2022-11-23T02:59:10.7598736Z Generating XML reports... 2022-11-23T02:59:10.7600519Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221123025908.xml 2022-11-23T02:59:10.7601553Z 2022-11-23T02:59:10.7602352Z ##[endgroup] 2022-11-23T02:59:10.7604148Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_hoizw_sj) 2022-11-23T02:59:10.7605161Z 2022-11-23T02:59:10.7605905Z Running distributed/fsdp/test_fsdp_fx ... [2022-11-23 02:59:10.757315] 2022-11-23T02:59:10.7607968Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:59:10.758192] 2022-11-23T02:59:14.2442310Z 2022-11-23T02:59:14.2443286Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx 2022-11-23T02:59:14.2445776Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_fx_tjy6uawo) 2022-11-23T02:59:14.2447886Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx 2022-11-23T02:59:14.2475053Z 2022-11-23T02:59:14.2475434Z Running tests... 2022-11-23T02:59:14.2476984Z ---------------------------------------------------------------------- 2022-11-23T02:59:14.2478092Z test_symbolic_tracing_outputs (__main__.TestSymbolicTracing) 2022-11-23T02:59:14.2480315Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.497s) 2022-11-23T02:59:14.2481408Z 2022-11-23T02:59:14.2482149Z ---------------------------------------------------------------------- 2022-11-23T02:59:14.2482999Z Ran 1 test in 0.498s 2022-11-23T02:59:14.2483390Z 2022-11-23T02:59:14.2483629Z OK (skipped=1) 2022-11-23T02:59:14.2484009Z 2022-11-23T02:59:14.2484294Z Generating XML reports... 2022-11-23T02:59:14.2486437Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123025912.xml 2022-11-23T02:59:14.2487378Z 2022-11-23T02:59:14.2488174Z ##[endgroup] 2022-11-23T02:59:14.2489786Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_fx_tjy6uawo) 2022-11-23T02:59:14.2490677Z 2022-11-23T02:59:14.2491492Z Running distributed/fsdp/test_fsdp_flatten_params ... [2022-11-23 02:59:14.245011] 2022-11-23T02:59:14.2493655Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:59:14.245897] 2022-11-23T02:59:17.7249162Z 2022-11-23T02:59:17.7250375Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_flatten_params 2022-11-23T02:59:17.7252761Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_fgy6bv_p) 2022-11-23T02:59:17.7255088Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params 2022-11-23T02:59:17.7255870Z 2022-11-23T02:59:17.7256194Z Running tests... 2022-11-23T02:59:17.7257760Z ---------------------------------------------------------------------- 2022-11-23T02:59:17.7258798Z test_empty_module (__main__.TestFlattenParams) 2022-11-23T02:59:17.7260792Z Tests flattening an empty module (i.e. one without any parameters). ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.485s) 2022-11-23T02:59:17.7262260Z test_flat_param_shard_metadata (__main__.TestFlattenParams) 2022-11-23T02:59:17.7264300Z Tests that ``FlatParameter`` shard metadata are computed as expected. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:59:17.7265744Z test_flatten_nothing (__main__.TestFlattenParams) 2022-11-23T02:59:17.7267719Z Tests that constructing a ``FlatParamHandle`` with no parameters ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:59:17.7269188Z test_numel_with_shared_params (__main__.TestFlattenParams) 2022-11-23T02:59:17.7271194Z Tests that numel is preserved after flattening when there are shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:59:17.7272667Z test_numel_without_shared_params (__main__.TestFlattenParams) 2022-11-23T02:59:17.7274698Z Tests that numel is preserved after flattening when there are no shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:59:17.7276711Z test_output_with_shared_params (__main__.TestFlattenParams) 2022-11-23T02:59:17.7278779Z Tests a forward pass after flattening when there are shared parameters ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:59:17.7280289Z test_output_without_shared_params (__main__.TestFlattenParams) 2022-11-23T02:59:17.7282310Z Tests a forward pass after flattening when there are no shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:59:17.7283718Z test_partial_flattening (__main__.TestFlattenParams) 2022-11-23T02:59:17.7285593Z Tests flattening some submodules but not others. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:59:17.7287028Z test_pnorm_after_step_with_shared_params (__main__.TestFlattenParams) 2022-11-23T02:59:17.7289094Z Tests for parameter Frobenius norm parity after an optimizer step when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:59:17.7290313Z 2022-11-23T02:59:17.7291068Z ---------------------------------------------------------------------- 2022-11-23T02:59:17.7291910Z Ran 9 tests in 0.492s 2022-11-23T02:59:17.7292319Z 2022-11-23T02:59:17.7292569Z OK (skipped=9) 2022-11-23T02:59:17.7292952Z 2022-11-23T02:59:17.7293258Z Generating XML reports... 2022-11-23T02:59:17.7294995Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20221123025915.xml 2022-11-23T02:59:17.7295969Z 2022-11-23T02:59:17.7296759Z ##[endgroup] 2022-11-23T02:59:17.7298517Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_fgy6bv_p) 2022-11-23T02:59:17.7299511Z 2022-11-23T02:59:17.7300261Z Running distributed/fsdp/test_fsdp_core ... [2022-11-23 02:59:17.725636] 2022-11-23T02:59:17.7302304Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:59:17.726518] 2022-11-23T03:03:54.2671322Z 2022-11-23T03:03:54.2672568Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_core 2022-11-23T03:03:54.2677276Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_core_ck6uj0_r) 2022-11-23T03:03:54.2680977Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core 2022-11-23T03:03:54.2681831Z 2022-11-23T03:03:54.2682158Z Running tests... 2022-11-23T03:03:54.2685987Z ---------------------------------------------------------------------- 2022-11-23T03:03:54.2687501Z test_pre_backward_hook_registration_after_state_dict (__main__.TestHooks) 2022-11-23T03:03:54.2690486Z Tests that FSDP pre-backward hooks are registered on forward pass ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.479s) 2022-11-23T03:03:54.2692482Z test_pre_backward_hook_registration_cuda_first_False (__main__.TestHooks) 2022-11-23T03:03:54.2695090Z Tests that FSDP pre-backward hooks are registered on forward pass ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2696966Z test_pre_backward_hook_registration_cuda_first_True (__main__.TestHooks) 2022-11-23T03:03:54.2699453Z Tests that FSDP pre-backward hooks are registered on forward pass ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2701291Z test_register_functions_called_cuda_first_False_mixed_precision_False (__main__.TestHooks) 2022-11-23T03:03:54.2703868Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2705790Z test_register_functions_called_cuda_first_False_mixed_precision_True (__main__.TestHooks) 2022-11-23T03:03:54.2708293Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2710233Z test_register_functions_called_cuda_first_True_mixed_precision_False (__main__.TestHooks) 2022-11-23T03:03:54.2712697Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2714594Z test_register_functions_called_cuda_first_True_mixed_precision_True (__main__.TestHooks) 2022-11-23T03:03:54.2717140Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2719911Z test_transformer_no_grad_mixed_precision_False (__main__.TestNoGrad) 2022-11-23T03:03:54.2722311Z Tests that for an FSDP-wrapped transformer model with shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2723984Z test_transformer_no_grad_mixed_precision_True (__main__.TestNoGrad) 2022-11-23T03:03:54.2726153Z Tests that for an FSDP-wrapped transformer model with shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2727800Z test_param_change_after_init_mixed_precision_False (__main__.TestParamInit) 2022-11-23T03:03:54.2730079Z Tests that changing FSDP model parameter values in-place after FSDP ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2731831Z test_param_change_after_init_mixed_precision_True (__main__.TestParamInit) 2022-11-23T03:03:54.2734077Z Tests that changing FSDP model parameter values in-place after FSDP ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2735789Z test_delayed_optim_step_offload_false_no_shard (__main__.TestParityWithDDP) 2022-11-23T03:03:54.2738148Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2739812Z test_delayed_optim_step_offload_false_none (__main__.TestParityWithDDP) 2022-11-23T03:03:54.2742017Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2743839Z test_delayed_optim_step_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T03:03:54.2746065Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.2747691Z test_delayed_optim_step_offload_true_no_shard (__main__.TestParityWithDDP) 2022-11-23T03:03:54.2749238Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13024 2022-11-23T03:03:54.2750808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13025 2022-11-23T03:03:54.2752658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.2754046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.2755790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.2757402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.2758832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:03:54.2760718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.2761969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.2763719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.2765133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.2766501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:03:54.2768507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.2770698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.2772420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:03:54.2773873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:03:54.2775227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2776687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2780534Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.2782906Z warnings.warn( 2022-11-23T03:03:54.2783764Z File "", line 1, in 2022-11-23T03:03:54.2784887Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2786009Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2787309Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2788440Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2789516Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2790531Z self.run() 2022-11-23T03:03:54.2791587Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2792688Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2794357Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2795528Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2797012Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2797353Z getattr(self, test_name)() 2022-11-23T03:03:54.2797856Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2798222Z fn() 2022-11-23T03:03:54.2798714Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2799101Z test(self, **param_kwargs) 2022-11-23T03:03:54.2799596Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2799975Z return func(*args, **kwargs) 2022-11-23T03:03:54.2800322Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2800682Z self.run_subtests( 2022-11-23T03:03:54.2801167Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2801557Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2802088Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2802496Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2803035Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2803387Z output = model(*input) 2022-11-23T03:03:54.2803903Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2804284Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2804813Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2805317Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2805871Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2806271Z _lazy_init(state, module) 2022-11-23T03:03:54.2806724Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2807115Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2807608Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2807968Z return func(*args, **kwargs) 2022-11-23T03:03:54.2808485Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2808850Z p_assert( 2022-11-23T03:03:54.2809261Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2809626Z traceback.print_stack() 2022-11-23T03:03:54.2810003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2811268Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.2811995Z warnings.warn( 2022-11-23T03:03:54.2812269Z File "", line 1, in 2022-11-23T03:03:54.2812634Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2812991Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2813308Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2813667Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2814039Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2814365Z self.run() 2022-11-23T03:03:54.2814693Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2815051Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2815546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2815878Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2816397Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2816774Z getattr(self, test_name)() 2022-11-23T03:03:54.2817273Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2817630Z fn() 2022-11-23T03:03:54.2818111Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2818560Z test(self, **param_kwargs) 2022-11-23T03:03:54.2819043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2819413Z return func(*args, **kwargs) 2022-11-23T03:03:54.2819793Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2820143Z self.run_subtests( 2022-11-23T03:03:54.2820683Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2821082Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2821557Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2822008Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2822531Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2822899Z output = model(*input) 2022-11-23T03:03:54.2823354Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2823713Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2824221Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2824601Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2825126Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2825489Z _lazy_init(state, module) 2022-11-23T03:03:54.2825965Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2826354Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2826836Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2827190Z return func(*args, **kwargs) 2022-11-23T03:03:54.2827693Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2828051Z p_assert( 2022-11-23T03:03:54.2828493Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2828847Z traceback.print_stack() 2022-11-23T03:03:54.2829214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2829577Z File "", line 1, in 2022-11-23T03:03:54.2829887Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2830240Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2830590Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2830940Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2831302Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2831621Z self.run() 2022-11-23T03:03:54.2831901Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2832247Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2832726Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2833086Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2833582Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2833952Z getattr(self, test_name)() 2022-11-23T03:03:54.2834434Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2834742Z fn() 2022-11-23T03:03:54.2835204Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2835573Z test(self, **param_kwargs) 2022-11-23T03:03:54.2836112Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2836480Z return func(*args, **kwargs) 2022-11-23T03:03:54.2836858Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2837166Z self.run_subtests( 2022-11-23T03:03:54.2837636Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2838031Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2838540Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2838978Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2839497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2839870Z output = model(*input) 2022-11-23T03:03:54.2840288Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2840649Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2841160Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2841579Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2842102Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2842471Z _lazy_init(state, module) 2022-11-23T03:03:54.2842951Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2843289Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2843772Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2844172Z return func(*args, **kwargs) 2022-11-23T03:03:54.2844677Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2845031Z p_assert( 2022-11-23T03:03:54.2845476Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2845793Z traceback.print_stack() 2022-11-23T03:03:54.2846164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2846524Z File "", line 1, in 2022-11-23T03:03:54.2846882Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2847229Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2847578Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2847926Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2848252Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2848567Z self.run() 2022-11-23T03:03:54.2848887Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2849230Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2849709Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2850072Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2850526Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2850897Z getattr(self, test_name)() 2022-11-23T03:03:54.2851378Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2851723Z fn() 2022-11-23T03:03:54.2852191Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2852558Z test(self, **param_kwargs) 2022-11-23T03:03:54.2853038Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2853365Z return func(*args, **kwargs) 2022-11-23T03:03:54.2853738Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2854084Z self.run_subtests( 2022-11-23T03:03:54.2854552Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2854997Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2855513Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2855866Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2856384Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2856747Z output = model(*input) 2022-11-23T03:03:54.2857191Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2857557Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2858064Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2858479Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2858965Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2859332Z _lazy_init(state, module) 2022-11-23T03:03:54.2859808Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2860183Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2860705Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2861063Z return func(*args, **kwargs) 2022-11-23T03:03:54.2861564Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2861880Z p_assert( 2022-11-23T03:03:54.2862325Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2862683Z traceback.print_stack() 2022-11-23T03:03:54.2863051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2863416Z File "", line 1, in 2022-11-23T03:03:54.2863766Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2864070Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2864418Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2864761Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2865125Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2865441Z self.run() 2022-11-23T03:03:54.2865759Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2866064Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2866543Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2866901Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2867388Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2867761Z getattr(self, test_name)() 2022-11-23T03:03:54.2868246Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2868587Z fn() 2022-11-23T03:03:54.2869011Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2869378Z test(self, **param_kwargs) 2022-11-23T03:03:54.2869860Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2870227Z return func(*args, **kwargs) 2022-11-23T03:03:54.2870601Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2870949Z self.run_subtests( 2022-11-23T03:03:54.2871415Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2871813Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2872328Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2872719Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2873234Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2873603Z output = model(*input) 2022-11-23T03:03:54.2874054Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2874378Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2874887Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2875306Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2875830Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2876255Z _lazy_init(state, module) 2022-11-23T03:03:54.2876793Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2877175Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2877619Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2877977Z return func(*args, **kwargs) 2022-11-23T03:03:54.2878481Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2878873Z p_assert( 2022-11-23T03:03:54.2879316Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2879671Z traceback.print_stack() 2022-11-23T03:03:54.2880046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2880363Z File "", line 1, in 2022-11-23T03:03:54.2880709Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2881062Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2881407Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2881752Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2882114Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2882395Z self.run() 2022-11-23T03:03:54.2882712Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2883060Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2883533Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2883896Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2884392Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2884718Z getattr(self, test_name)() 2022-11-23T03:03:54.2885195Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2885537Z fn() 2022-11-23T03:03:54.2885995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2886361Z test(self, **param_kwargs) 2022-11-23T03:03:54.2886836Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2887322Z return func(*args, **kwargs) 2022-11-23T03:03:54.2887648Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2888040Z self.run_subtests( 2022-11-23T03:03:54.2888499Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2888882Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2889383Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2889760Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2890259Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2890577Z output = model(*input) 2022-11-23T03:03:54.2891016Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2891376Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2891873Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2892282Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2892792Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2893151Z _lazy_init(state, module) 2022-11-23T03:03:54.2893612Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2893983Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2894456Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2894802Z return func(*args, **kwargs) 2022-11-23T03:03:54.2895293Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2895638Z p_assert( 2022-11-23T03:03:54.2896033Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2896380Z traceback.print_stack() 2022-11-23T03:03:54.2896738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2897089Z File "", line 1, in 2022-11-23T03:03:54.2897432Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2897766Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2898105Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2898406Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2898758Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2899070Z self.run() 2022-11-23T03:03:54.2899381Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2899721Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2900189Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2900499Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2900983Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2901341Z getattr(self, test_name)() 2022-11-23T03:03:54.2901812Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2902149Z fn() 2022-11-23T03:03:54.2902599Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2902919Z test(self, **param_kwargs) 2022-11-23T03:03:54.2903543Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2903897Z return func(*args, **kwargs) 2022-11-23T03:03:54.2904301Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2904641Z self.run_subtests( 2022-11-23T03:03:54.2905101Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2905494Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2905956Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2906329Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2906829Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2907187Z output = model(*input) 2022-11-23T03:03:54.2907624Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2907977Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2908476Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2908843Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2909476Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2909838Z _lazy_init(state, module) 2022-11-23T03:03:54.2910306Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2910679Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2911151Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2911498Z return func(*args, **kwargs) 2022-11-23T03:03:54.2911947Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2912295Z p_assert( 2022-11-23T03:03:54.2912725Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2913069Z traceback.print_stack() 2022-11-23T03:03:54.2913428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2913778Z File "", line 1, in 2022-11-23T03:03:54.2914082Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2914421Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2914759Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2915099Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2915450Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2915761Z self.run() 2022-11-23T03:03:54.2916034Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2916619Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2917106Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2917467Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2917962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2918329Z getattr(self, test_name)() 2022-11-23T03:03:54.2918808Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2919116Z fn() 2022-11-23T03:03:54.2919589Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2920029Z test(self, **param_kwargs) 2022-11-23T03:03:54.2920512Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2920925Z return func(*args, **kwargs) 2022-11-23T03:03:54.2921297Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2921604Z self.run_subtests( 2022-11-23T03:03:54.2922076Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2922473Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2922988Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2923372Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2923884Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2924248Z output = model(*input) 2022-11-23T03:03:54.2924662Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2925024Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2925531Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2925949Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2926516Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2926881Z _lazy_init(state, module) 2022-11-23T03:03:54.2927479Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2927808Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2954792Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2955154Z return func(*args, **kwargs) 2022-11-23T03:03:54.2955679Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2956123Z p_assert( 2022-11-23T03:03:54.2956736Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2957098Z traceback.print_stack() 2022-11-23T03:03:54.2957477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2957802Z File "", line 1, in 2022-11-23T03:03:54.2958171Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2958521Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2958871Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2959293Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2959654Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2959936Z self.run() 2022-11-23T03:03:54.2960314Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2960660Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2961131Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2961478Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2961968Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2962285Z getattr(self, test_name)() 2022-11-23T03:03:54.2962776Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2963096Z fn() 2022-11-23T03:03:54.2963564Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2963937Z test(self, **param_kwargs) 2022-11-23T03:03:54.2964534Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2964906Z return func(*args, **kwargs) 2022-11-23T03:03:54.2965244Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2965599Z self.run_subtests( 2022-11-23T03:03:54.2966070Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2966466Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2966983Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2967374Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2967888Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2968221Z output = model(*input) 2022-11-23T03:03:54.2968667Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2969033Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2969592Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2970028Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2970560Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2970934Z _lazy_init(state, module) 2022-11-23T03:03:54.2971373Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2971750Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2972356Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2972706Z return func(*args, **kwargs) 2022-11-23T03:03:54.2973199Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2973549Z p_assert( 2022-11-23T03:03:54.2973944Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2974290Z traceback.print_stack() 2022-11-23T03:03:54.2974649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2975003Z File "", line 1, in 2022-11-23T03:03:54.2975347Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2975676Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2976020Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2976327Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2976673Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2976971Z self.run() 2022-11-23T03:03:54.2977285Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2977593Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2978000Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2978284Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2978706Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2979022Z getattr(self, test_name)() 2022-11-23T03:03:54.2979462Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2979798Z fn() 2022-11-23T03:03:54.2980251Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2980618Z test(self, **param_kwargs) 2022-11-23T03:03:54.2981089Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2981447Z return func(*args, **kwargs) 2022-11-23T03:03:54.2981823Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2982164Z self.run_subtests( 2022-11-23T03:03:54.2982608Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2982993Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2983456Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.2983836Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.2984337Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.2984686Z output = model(*input) 2022-11-23T03:03:54.2985127Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.2985480Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.2986010Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.2986384Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.2986840Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.2987156Z _lazy_init(state, module) 2022-11-23T03:03:54.2987606Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.2987978Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.2988454Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.2988801Z return func(*args, **kwargs) 2022-11-23T03:03:54.2989254Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.2989589Z p_assert( 2022-11-23T03:03:54.2990006Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.2990353Z traceback.print_stack() 2022-11-23T03:03:54.2990667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.2990969Z File "", line 1, in 2022-11-23T03:03:54.2991261Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.2991587Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.2991928Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.2992268Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.2992609Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.2992918Z self.run() 2022-11-23T03:03:54.2993175Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.2993486Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.2993938Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.2994274Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.2994755Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.2995113Z getattr(self, test_name)() 2022-11-23T03:03:54.2995584Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.2995877Z fn() 2022-11-23T03:03:54.2996589Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.2996962Z test(self, **param_kwargs) 2022-11-23T03:03:54.2997414Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.2997780Z return func(*args, **kwargs) 2022-11-23T03:03:54.2998158Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.2998469Z self.run_subtests( 2022-11-23T03:03:54.2998925Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.2999303Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.2999811Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3000187Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3000702Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3001066Z output = model(*input) 2022-11-23T03:03:54.3001521Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3001888Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3002398Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3002815Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3003344Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3003696Z _lazy_init(state, module) 2022-11-23T03:03:54.3004169Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3004512Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3004996Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3005354Z return func(*args, **kwargs) 2022-11-23T03:03:54.3005837Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3006193Z p_assert( 2022-11-23T03:03:54.3006635Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3006973Z traceback.print_stack() 2022-11-23T03:03:54.3007418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3007768Z File "", line 1, in 2022-11-23T03:03:54.3008081Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3008423Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3008770Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3009099Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3009403Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3009713Z self.run() 2022-11-23T03:03:54.3010004Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3010328Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3010793Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3011142Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3011592Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3011938Z getattr(self, test_name)() 2022-11-23T03:03:54.3012412Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3012806Z fn() 2022-11-23T03:03:54.3013246Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3013606Z test(self, **param_kwargs) 2022-11-23T03:03:54.3014054Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3014362Z return func(*args, **kwargs) 2022-11-23T03:03:54.3014729Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3015026Z self.run_subtests( 2022-11-23T03:03:54.3015468Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3015835Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3016281Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3016610Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3017045Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3017350Z output = model(*input) 2022-11-23T03:03:54.3017776Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3018096Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3018549Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3018913Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3019364Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3019686Z _lazy_init(state, module) 2022-11-23T03:03:54.3020106Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3020436Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3020867Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3021176Z return func(*args, **kwargs) 2022-11-23T03:03:54.3021627Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3021912Z p_assert( 2022-11-23T03:03:54.3022290Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3022584Z traceback.print_stack() 2022-11-23T03:03:54.3022888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3023187Z File "", line 1, in 2022-11-23T03:03:54.3023476Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3023753Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3024040Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3024325Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3024629Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3024885Z self.run() 2022-11-23T03:03:54.3025143Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3025428Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3025826Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3026123Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3026550Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3026889Z getattr(self, test_name)() 2022-11-23T03:03:54.3027305Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3027587Z fn() 2022-11-23T03:03:54.3027970Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3028276Z test(self, **param_kwargs) 2022-11-23T03:03:54.3028686Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3028987Z return func(*args, **kwargs) 2022-11-23T03:03:54.3029301Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3029584Z self.run_subtests( 2022-11-23T03:03:54.3029985Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3030300Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3030743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3031068Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3031552Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3031860Z output = model(*input) 2022-11-23T03:03:54.3032243Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3032541Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3032973Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3033324Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3033776Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3034086Z _lazy_init(state, module) 2022-11-23T03:03:54.3034496Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3034809Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3035207Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3035491Z return func(*args, **kwargs) 2022-11-23T03:03:54.3035914Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3036247Z p_assert( 2022-11-23T03:03:54.3036795Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3037078Z traceback.print_stack() 2022-11-23T03:03:54.3037381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3037670Z File "", line 1, in 2022-11-23T03:03:54.3037953Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3038293Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3038572Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3038857Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3039155Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3039401Z self.run() 2022-11-23T03:03:54.3039650Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3039925Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3040337Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3040632Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3041054Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3041401Z getattr(self, test_name)() 2022-11-23T03:03:54.3041808Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3042080Z fn() 2022-11-23T03:03:54.3042473Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3042773Z test(self, **param_kwargs) 2022-11-23T03:03:54.3043177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3043475Z return func(*args, **kwargs) 2022-11-23T03:03:54.3043793Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3044071Z self.run_subtests( 2022-11-23T03:03:54.3044469Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3044798Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3045243Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3045599Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3046048Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3046340Z output = model(*input) 2022-11-23T03:03:54.3046719Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3047012Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3047660Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3047983Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3048405Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3048678Z _lazy_init(state, module) 2022-11-23T03:03:54.3049045Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3049329Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3049709Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3049973Z return func(*args, **kwargs) 2022-11-23T03:03:54.3050371Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3050636Z p_assert( 2022-11-23T03:03:54.3050976Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3051235Z traceback.print_stack() 2022-11-23T03:03:54.3051510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3051775Z File "", line 1, in 2022-11-23T03:03:54.3052033Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3052286Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3052547Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3052801Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3053072Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3053300Z self.run() 2022-11-23T03:03:54.3053533Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3053786Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3054162Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3054466Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3054858Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3055134Z getattr(self, test_name)() 2022-11-23T03:03:54.3055517Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3055772Z fn() 2022-11-23T03:03:54.3056128Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3056403Z test(self, **param_kwargs) 2022-11-23T03:03:54.3056778Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3057047Z return func(*args, **kwargs) 2022-11-23T03:03:54.3057326Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3057588Z self.run_subtests( 2022-11-23T03:03:54.3057954Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3058252Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3058686Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3058979Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3059391Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3059668Z output = model(*input) 2022-11-23T03:03:54.3060020Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3060289Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3060693Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3061016Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3061433Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3061705Z _lazy_init(state, module) 2022-11-23T03:03:54.3062077Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3062363Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3062741Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3063004Z return func(*args, **kwargs) 2022-11-23T03:03:54.3063395Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3063655Z p_assert( 2022-11-23T03:03:54.3063996Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3064265Z traceback.print_stack() 2022-11-23T03:03:54.3064545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3064812Z File "", line 1, in 2022-11-23T03:03:54.3065070Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3065327Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3065586Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3065841Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3066112Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3066341Z self.run() 2022-11-23T03:03:54.3066565Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3066821Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3067257Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3067525Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3067917Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3068192Z getattr(self, test_name)() 2022-11-23T03:03:54.3068570Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3068818Z fn() 2022-11-23T03:03:54.3069182Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3069459Z test(self, **param_kwargs) 2022-11-23T03:03:54.3069836Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3070111Z return func(*args, **kwargs) 2022-11-23T03:03:54.3070394Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3070655Z self.run_subtests( 2022-11-23T03:03:54.3071023Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3071351Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3071763Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3073173Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3073589Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3073866Z output = model(*input) 2022-11-23T03:03:54.3074218Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3074483Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3074890Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3075208Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3075627Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3075902Z _lazy_init(state, module) 2022-11-23T03:03:54.3076500Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3076811Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3077220Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3077510Z return func(*args, **kwargs) 2022-11-23T03:03:54.3077946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3078237Z p_assert( 2022-11-23T03:03:54.3078618Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3078940Z traceback.print_stack() 2022-11-23T03:03:54.3079235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3079527Z File "", line 1, in 2022-11-23T03:03:54.3079806Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3080086Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3080368Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3080651Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3080940Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3081189Z self.run() 2022-11-23T03:03:54.3081439Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3081771Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3082183Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3082478Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3082908Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3083202Z getattr(self, test_name)() 2022-11-23T03:03:54.3083613Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3083889Z fn() 2022-11-23T03:03:54.3084282Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3084581Z test(self, **param_kwargs) 2022-11-23T03:03:54.3084991Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3085286Z return func(*args, **kwargs) 2022-11-23T03:03:54.3085594Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3085873Z self.run_subtests( 2022-11-23T03:03:54.3086314Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3086641Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3087087Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3087512Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3087916Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3088191Z output = model(*input) 2022-11-23T03:03:54.3088542Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3088821Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3089237Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3089562Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3089985Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3090254Z _lazy_init(state, module) 2022-11-23T03:03:54.3090631Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3090915Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3091295Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3091562Z return func(*args, **kwargs) 2022-11-23T03:03:54.3091963Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3092233Z p_assert( 2022-11-23T03:03:54.3092573Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3092838Z traceback.print_stack() 2022-11-23T03:03:54.3093121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3093393Z File "", line 1, in 2022-11-23T03:03:54.3093659Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3093921Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3094175Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3094432Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3094706Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3094975Z self.run() 2022-11-23T03:03:54.3095210Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3095470Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3095845Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3096123Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3096514Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3096789Z getattr(self, test_name)() 2022-11-23T03:03:54.3097173Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3097430Z fn() 2022-11-23T03:03:54.3097799Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3098071Z test(self, **param_kwargs) 2022-11-23T03:03:54.3098451Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3098726Z return func(*args, **kwargs) 2022-11-23T03:03:54.3099009Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3099271Z self.run_subtests( 2022-11-23T03:03:54.3099679Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3099975Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3100392Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3100693Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3101104Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3101383Z output = model(*input) 2022-11-23T03:03:54.3101744Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3102018Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3102420Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3102746Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3103164Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3103440Z _lazy_init(state, module) 2022-11-23T03:03:54.3103818Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3104106Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3104492Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3104754Z return func(*args, **kwargs) 2022-11-23T03:03:54.3105157Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3105424Z p_assert( 2022-11-23T03:03:54.3105775Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3106044Z traceback.print_stack() 2022-11-23T03:03:54.3106323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3106589Z File "", line 1, in 2022-11-23T03:03:54.3106849Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3107109Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3107371Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3107630Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3107938Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3108173Z self.run() 2022-11-23T03:03:54.3108402Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3108657Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3109038Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3109313Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3109703Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3109983Z getattr(self, test_name)() 2022-11-23T03:03:54.3110360Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3110621Z fn() 2022-11-23T03:03:54.3110987Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3111265Z test(self, **param_kwargs) 2022-11-23T03:03:54.3111640Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3111915Z return func(*args, **kwargs) 2022-11-23T03:03:54.3112239Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3112493Z self.run_subtests( 2022-11-23T03:03:54.3112865Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3113167Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3113578Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3113878Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3114295Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3114576Z output = model(*input) 2022-11-23T03:03:54.3114921Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3115194Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3115603Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3115931Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3116611Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3116924Z _lazy_init(state, module) 2022-11-23T03:03:54.3117331Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3117645Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3118065Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3118356Z return func(*args, **kwargs) 2022-11-23T03:03:54.3118790Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3119085Z p_assert( 2022-11-23T03:03:54.3119464Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3119748Z traceback.print_stack() 2022-11-23T03:03:54.3120050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3120344Z File "", line 1, in 2022-11-23T03:03:54.3120630Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3120911Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3121194Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3121514Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3121817Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3122070Z self.run() 2022-11-23T03:03:54.3122325Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3122608Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3123026Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3123319Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3123741Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3124042Z getattr(self, test_name)() 2022-11-23T03:03:54.3124459Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3124731Z fn() 2022-11-23T03:03:54.3125130Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3125432Z test(self, **param_kwargs) 2022-11-23T03:03:54.3125840Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3126184Z return func(*args, **kwargs) 2022-11-23T03:03:54.3126494Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3126775Z self.run_subtests( 2022-11-23T03:03:54.3127401Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3127702Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3128118Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3128410Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3128825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3129104Z output = model(*input) 2022-11-23T03:03:54.3129457Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3129731Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3130138Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3130468Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3130884Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3131162Z _lazy_init(state, module) 2022-11-23T03:03:54.3131537Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3131824Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3132210Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3132479Z return func(*args, **kwargs) 2022-11-23T03:03:54.3132875Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3133141Z p_assert( 2022-11-23T03:03:54.3133492Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3133764Z traceback.print_stack() 2022-11-23T03:03:54.3134043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3134316Z File "", line 1, in 2022-11-23T03:03:54.3134580Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3134834Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3135154Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3135414Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3135690Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3135922Z self.run() 2022-11-23T03:03:54.3136159Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3136410Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3136795Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3137068Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3137462Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3137742Z getattr(self, test_name)() 2022-11-23T03:03:54.3138127Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3138382Z fn() 2022-11-23T03:03:54.3138739Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3139018Z test(self, **param_kwargs) 2022-11-23T03:03:54.3139436Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3139713Z return func(*args, **kwargs) 2022-11-23T03:03:54.3139994Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3140258Z self.run_subtests( 2022-11-23T03:03:54.3140628Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3140928Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3141333Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3141630Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3142043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3142320Z output = model(*input) 2022-11-23T03:03:54.3142679Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3142945Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3143354Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3143681Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3144099Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3144379Z _lazy_init(state, module) 2022-11-23T03:03:54.3144764Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3145054Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3145430Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3145701Z return func(*args, **kwargs) 2022-11-23T03:03:54.3146107Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3146375Z p_assert( 2022-11-23T03:03:54.3146724Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3146992Z traceback.print_stack() 2022-11-23T03:03:54.3147263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3147537Z File "", line 1, in 2022-11-23T03:03:54.3147803Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3148095Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3148362Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3148624Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3148904Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3149134Z self.run() 2022-11-23T03:03:54.3149368Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3149626Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3150010Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3150282Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3150676Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3150951Z getattr(self, test_name)() 2022-11-23T03:03:54.3151336Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3151595Z fn() 2022-11-23T03:03:54.3151959Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3152269Z test(self, **param_kwargs) 2022-11-23T03:03:54.3152658Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3152938Z return func(*args, **kwargs) 2022-11-23T03:03:54.3153215Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3153475Z self.run_subtests( 2022-11-23T03:03:54.3153850Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3154149Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3154565Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3154864Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3155272Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3155549Z output = model(*input) 2022-11-23T03:03:54.3155905Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3156463Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3156946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3157293Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3157746Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3158039Z _lazy_init(state, module) 2022-11-23T03:03:54.3158438Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3158742Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3159149Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3159434Z return func(*args, **kwargs) 2022-11-23T03:03:54.3159863Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3160151Z p_assert( 2022-11-23T03:03:54.3160515Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3160800Z traceback.print_stack() 2022-11-23T03:03:54.3161097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3161438Z File "", line 1, in 2022-11-23T03:03:54.3161718Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3161996Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3162270Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3162550Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3162841Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3163090Z self.run() 2022-11-23T03:03:54.3163341Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3163618Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3164014Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3164300Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3164716Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3165013Z getattr(self, test_name)() 2022-11-23T03:03:54.3165417Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3165691Z fn() 2022-11-23T03:03:54.3166137Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3166427Z test(self, **param_kwargs) 2022-11-23T03:03:54.3166835Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3167234Z return func(*args, **kwargs) 2022-11-23T03:03:54.3167518Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3167779Z self.run_subtests( 2022-11-23T03:03:54.3168154Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3168459Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3168866Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3169164Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3169577Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3169857Z output = model(*input) 2022-11-23T03:03:54.3170215Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3170491Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3170896Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3171225Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3171650Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3171927Z _lazy_init(state, module) 2022-11-23T03:03:54.3172304Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3172596Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3172979Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3173244Z return func(*args, **kwargs) 2022-11-23T03:03:54.3173645Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3173914Z p_assert( 2022-11-23T03:03:54.3174260Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3174529Z traceback.print_stack() 2022-11-23T03:03:54.3174844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3175116Z File "", line 1, in 2022-11-23T03:03:54.3175371Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3175629Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3175773Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3175874Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3176022Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3176089Z self.run() 2022-11-23T03:03:54.3176229Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3176328Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3176586Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3176679Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3176956Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3177039Z getattr(self, test_name)() 2022-11-23T03:03:54.3177344Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3177399Z fn() 2022-11-23T03:03:54.3177678Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3177759Z test(self, **param_kwargs) 2022-11-23T03:03:54.3178026Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3178107Z return func(*args, **kwargs) 2022-11-23T03:03:54.3178282Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:03:54.3178356Z self.run_subtests( 2022-11-23T03:03:54.3178623Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3178734Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3179013Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3179118Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3179407Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3179483Z output = model(*input) 2022-11-23T03:03:54.3179729Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3179822Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3180109Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3180232Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3180510Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3180589Z _lazy_init(state, module) 2022-11-23T03:03:54.3180851Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3180948Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3181204Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3181283Z return func(*args, **kwargs) 2022-11-23T03:03:54.3181573Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3181639Z p_assert( 2022-11-23T03:03:54.3181897Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3182012Z traceback.print_stack() 2022-11-23T03:03:54.3182181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3182347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3182508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3182670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3182829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3190698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3190919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3191086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3191260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3191422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3191588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3191828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3191987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3192149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3192310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3192474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3192632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3192791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3192948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3193108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3193268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3193421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3193575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3193733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3193887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3194045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3194202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3194353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3194511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3194663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3194818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3194971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3195117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3195275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3195471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3195630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3195787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3195944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3196156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3197015Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:03:54.3197644Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:03:54.3197824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3198042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3198211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3198383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3198554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3198725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3198895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3199067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3199147Z dist init r=0, world=2 2022-11-23T03:03:54.3199409Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3199663Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3199926Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3200175Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3200420Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3200678Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3200925Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3201166Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3201418Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3201660Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3201939Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3202194Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3202275Z dist init r=1, world=2 2022-11-23T03:03:54.3202517Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3202752Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3202997Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3203239Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3203512Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3203752Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3203994Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3204229Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3204478Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3204718Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3204961Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3205205Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3205276Z ok (41.620s) 2022-11-23T03:03:54.3205433Z test_delayed_optim_step_offload_true_none (__main__.TestParityWithDDP) 2022-11-23T03:03:54.3205871Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T03:03:54.3206042Z test_delayed_optim_step_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T03:03:54.3206472Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:54.3206640Z test_delayed_reduce_scatter_offload_false_no_shard (__main__.TestParityWithDDP) 2022-11-23T03:03:54.3207175Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:54.3207433Z test_delayed_reduce_scatter_offload_false_none (__main__.TestParityWithDDP) 2022-11-23T03:03:54.3207658Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13161 2022-11-23T03:03:54.3207846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13162 2022-11-23T03:03:54.3208131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.3208254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.3208550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.3208683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.3208859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:03:54.3209143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.3209266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.3209555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.3209688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.3209863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:03:54.3210199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.3210504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.3210665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:03:54.3210813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:03:54.3210979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3211145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3211959Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.3212034Z warnings.warn( 2022-11-23T03:03:54.3212198Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3212995Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.3213071Z warnings.warn( 2022-11-23T03:03:54.3213238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3213404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3213564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3213889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3214220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3214390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3214560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3214770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3214940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3215114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3215286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3215460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3215631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3215801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3215970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3216135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3216309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3216479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3216681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3217550Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:314.) 2022-11-23T03:03:54.3217730Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T03:03:54.3217902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3218756Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:314.) 2022-11-23T03:03:54.3218936Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T03:03:54.3219114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3219281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3219456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3219633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3219805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3219975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3220147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3220322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3220492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3220664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3220837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3221006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3221218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3221384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3221556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3221728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3221898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3222067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3222233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3222399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3222571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3222745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3222909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3223108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3223279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3223357Z dist init r=1, world=2 2022-11-23T03:03:54.3223429Z dist init r=0, world=2 2022-11-23T03:03:54.3223496Z ok (59.753s) 2022-11-23T03:03:54.3223668Z test_delayed_reduce_scatter_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T03:03:54.3223914Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13298 2022-11-23T03:03:54.3224082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13299 2022-11-23T03:03:54.3224394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.3224527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.3224843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.3224988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.3225399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:03:54.3225679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.3225800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.3226091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.3226226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.3226400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:03:54.3226700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.3227003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.3227163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:03:54.3227325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:03:54.3227488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3227648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3228489Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.3228564Z warnings.warn( 2022-11-23T03:03:54.3228731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3229532Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.3229606Z warnings.warn( 2022-11-23T03:03:54.3229772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3229968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3230129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3230289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3230448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3230606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3230768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3230928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3231090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3231251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3231410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3231571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3231729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3231888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3232044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3232198Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3232358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3232510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3232668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3233451Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:314.) 2022-11-23T03:03:54.3233614Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T03:03:54.3233774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3234596Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:314.) 2022-11-23T03:03:54.3234766Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T03:03:54.3234922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3235083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3235243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3235402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3235561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3235719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3235923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3236137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3236464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3236630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3236796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3236963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3237135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3237300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3237463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3237630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3237798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3237961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3238127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3238291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3238447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3238620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3238788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3238950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3239113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3239189Z dist init r=0, world=2 2022-11-23T03:03:54.3239264Z dist init r=1, world=2 2022-11-23T03:03:54.3239332Z ok (62.678s) 2022-11-23T03:03:54.3239495Z test_delayed_reduce_scatter_offload_true_no_shard (__main__.TestParityWithDDP) 2022-11-23T03:03:54.3239917Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T03:03:54.3240072Z test_delayed_reduce_scatter_offload_true_none (__main__.TestParityWithDDP) 2022-11-23T03:03:54.3240353Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13435 2022-11-23T03:03:54.3240516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13436 2022-11-23T03:03:54.3240818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.3240946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.3241253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.3241391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.3241576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:03:54.3241873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.3242005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.3242303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.3242480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.3242664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:03:54.3242988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.3243307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.3243480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:03:54.3243648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:03:54.3243826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3243999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3244845Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.3244924Z warnings.warn( 2022-11-23T03:03:54.3245018Z File "", line 1, in 2022-11-23T03:03:54.3245179Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3245283Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3245433Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3245541Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3245699Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3245772Z self.run() 2022-11-23T03:03:54.3245925Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3246030Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3246303Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3246397Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3246682Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3246768Z getattr(self, test_name)() 2022-11-23T03:03:54.3247061Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3247162Z fn() 2022-11-23T03:03:54.3247555Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3247634Z test(self, **param_kwargs) 2022-11-23T03:03:54.3247909Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3247991Z return func(*args, **kwargs) 2022-11-23T03:03:54.3248168Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3248246Z self.run_subtests( 2022-11-23T03:03:54.3248517Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3248628Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3248904Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3249007Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3249293Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3249403Z output = model(*input) 2022-11-23T03:03:54.3249656Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3249749Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3250032Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3250153Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3250431Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3250510Z _lazy_init(state, module) 2022-11-23T03:03:54.3250781Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3250875Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3251128Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3251215Z return func(*args, **kwargs) 2022-11-23T03:03:54.3251506Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3251572Z p_assert( 2022-11-23T03:03:54.3251831Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3251911Z traceback.print_stack() 2022-11-23T03:03:54.3252079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3252882Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.3252957Z warnings.warn( 2022-11-23T03:03:54.3253044Z File "", line 1, in 2022-11-23T03:03:54.3253195Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3253289Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3253427Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3253527Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3253668Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3253772Z self.run() 2022-11-23T03:03:54.3253915Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3254017Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3254277Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3254372Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3254649Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3254729Z getattr(self, test_name)() 2022-11-23T03:03:54.3255007Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3255076Z fn() 2022-11-23T03:03:54.3255359Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3255444Z test(self, **param_kwargs) 2022-11-23T03:03:54.3255723Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3255805Z return func(*args, **kwargs) 2022-11-23T03:03:54.3255985Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3256095Z self.run_subtests( 2022-11-23T03:03:54.3256376Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3256489Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3256758Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3256862Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3257153Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3257236Z output = model(*input) 2022-11-23T03:03:54.3257487Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3257582Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3257872Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3257998Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3258278Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3258359Z _lazy_init(state, module) 2022-11-23T03:03:54.3258624Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3258721Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3258976Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3259059Z return func(*args, **kwargs) 2022-11-23T03:03:54.3259345Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3259408Z p_assert( 2022-11-23T03:03:54.3259666Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3259749Z traceback.print_stack() 2022-11-23T03:03:54.3259910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3259999Z File "", line 1, in 2022-11-23T03:03:54.3260146Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3260243Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3260383Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3260486Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3260669Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3260733Z self.run() 2022-11-23T03:03:54.3260873Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3260972Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3261234Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3261325Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3261601Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3261685Z getattr(self, test_name)() 2022-11-23T03:03:54.3261962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3262020Z fn() 2022-11-23T03:03:54.3262297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3262371Z test(self, **param_kwargs) 2022-11-23T03:03:54.3262640Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3262722Z return func(*args, **kwargs) 2022-11-23T03:03:54.3262933Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3263007Z self.run_subtests( 2022-11-23T03:03:54.3263273Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3263382Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3263657Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3263763Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3264045Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3264127Z output = model(*input) 2022-11-23T03:03:54.3264374Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3264468Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3264757Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3264878Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3265162Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3265238Z _lazy_init(state, module) 2022-11-23T03:03:54.3265505Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3265593Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3265855Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3265937Z return func(*args, **kwargs) 2022-11-23T03:03:54.3266228Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3266291Z p_assert( 2022-11-23T03:03:54.3266546Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3266627Z traceback.print_stack() 2022-11-23T03:03:54.3266794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3266879Z File "", line 1, in 2022-11-23T03:03:54.3267027Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3267120Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3267260Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3267399Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3267548Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3267613Z self.run() 2022-11-23T03:03:54.3267756Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3267854Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3268101Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3268189Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3268465Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3268543Z getattr(self, test_name)() 2022-11-23T03:03:54.3268815Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3268873Z fn() 2022-11-23T03:03:54.3269147Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3269225Z test(self, **param_kwargs) 2022-11-23T03:03:54.3269495Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3269606Z return func(*args, **kwargs) 2022-11-23T03:03:54.3269785Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3269854Z self.run_subtests( 2022-11-23T03:03:54.3270119Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3270229Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3270501Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3270603Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3270890Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3270963Z output = model(*input) 2022-11-23T03:03:54.3271204Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3271295Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3271579Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3271696Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3271972Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3272047Z _lazy_init(state, module) 2022-11-23T03:03:54.3272311Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3272408Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3272661Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3272739Z return func(*args, **kwargs) 2022-11-23T03:03:54.3273026Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3273088Z p_assert( 2022-11-23T03:03:54.3273339Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3273419Z traceback.print_stack() 2022-11-23T03:03:54.3273583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3273665Z File "", line 1, in 2022-11-23T03:03:54.3273808Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3273902Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3274070Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3274172Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3274317Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3274381Z self.run() 2022-11-23T03:03:54.3274519Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3274613Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3274869Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3274953Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3275225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3275302Z getattr(self, test_name)() 2022-11-23T03:03:54.3275572Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3275632Z fn() 2022-11-23T03:03:54.3275908Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3275982Z test(self, **param_kwargs) 2022-11-23T03:03:54.3276525Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3276609Z return func(*args, **kwargs) 2022-11-23T03:03:54.3276792Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3276860Z self.run_subtests( 2022-11-23T03:03:54.3277144Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3277260Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3277548Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3277656Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3277950Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3278027Z output = model(*input) 2022-11-23T03:03:54.3278289Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3278385Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3278684Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3278806Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3279098Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3279178Z _lazy_init(state, module) 2022-11-23T03:03:54.3279460Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3279558Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3279823Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3279907Z return func(*args, **kwargs) 2022-11-23T03:03:54.3280207Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3280267Z p_assert( 2022-11-23T03:03:54.3280531Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3280613Z traceback.print_stack() 2022-11-23T03:03:54.3280784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3280871Z File "", line 1, in 2022-11-23T03:03:54.3281021Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3281158Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3281307Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3281411Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3281567Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3281633Z self.run() 2022-11-23T03:03:54.3281778Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3281878Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3282145Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3282234Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3282518Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3282601Z getattr(self, test_name)() 2022-11-23T03:03:54.3282881Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3282943Z fn() 2022-11-23T03:03:54.3283266Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3283348Z test(self, **param_kwargs) 2022-11-23T03:03:54.3283634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3283718Z return func(*args, **kwargs) 2022-11-23T03:03:54.3283902Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3283975Z self.run_subtests( 2022-11-23T03:03:54.3284252Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3284365Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3284660Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3284765Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3285063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3285141Z output = model(*input) 2022-11-23T03:03:54.3285395Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3285491Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3285789Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3285914Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3286201Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3286283Z _lazy_init(state, module) 2022-11-23T03:03:54.3286561Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3286658Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3286926Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3287008Z return func(*args, **kwargs) 2022-11-23T03:03:54.3287411Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3287472Z p_assert( 2022-11-23T03:03:54.3287724Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3287803Z traceback.print_stack() 2022-11-23T03:03:54.3287968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3288100Z File "", line 1, in 2022-11-23T03:03:54.3288243Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3288335Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3288473Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3288573Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3288717Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3288775Z self.run() 2022-11-23T03:03:54.3288913Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3289007Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3289261Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3289345Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3289621Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3289701Z getattr(self, test_name)() 2022-11-23T03:03:54.3289971Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3290028Z fn() 2022-11-23T03:03:54.3290360Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3290437Z test(self, **param_kwargs) 2022-11-23T03:03:54.3290705Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3290783Z return func(*args, **kwargs) 2022-11-23T03:03:54.3290957Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3291025Z self.run_subtests( 2022-11-23T03:03:54.3291287Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3291396Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3291669Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3291766Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3292049Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3292124Z output = model(*input) 2022-11-23T03:03:54.3292368Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3292459Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3292743Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3292861Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3293136Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3293210Z _lazy_init(state, module) 2022-11-23T03:03:54.3293476Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3293571Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3293824Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3293902Z return func(*args, **kwargs) 2022-11-23T03:03:54.3294188Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3294247Z p_assert( 2022-11-23T03:03:54.3294498Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3294576Z traceback.print_stack() 2022-11-23T03:03:54.3294775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3294854Z File "", line 1, in 2022-11-23T03:03:54.3294996Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3295088Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3295226Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3295325Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3295472Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3295533Z self.run() 2022-11-23T03:03:54.3295668Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3295764Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3296020Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3296107Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3296379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3296456Z getattr(self, test_name)() 2022-11-23T03:03:54.3296759Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3296817Z fn() 2022-11-23T03:03:54.3297091Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3297167Z test(self, **param_kwargs) 2022-11-23T03:03:54.3297430Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3297508Z return func(*args, **kwargs) 2022-11-23T03:03:54.3297681Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3297753Z self.run_subtests( 2022-11-23T03:03:54.3298016Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3298126Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3298400Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3298502Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3298786Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3298859Z output = model(*input) 2022-11-23T03:03:54.3299104Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3299195Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3299477Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3299598Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3299875Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3299949Z _lazy_init(state, module) 2022-11-23T03:03:54.3300214Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3300305Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3300552Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3300632Z return func(*args, **kwargs) 2022-11-23T03:03:54.3300916Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3300975Z p_assert( 2022-11-23T03:03:54.3301227Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3301338Z traceback.print_stack() 2022-11-23T03:03:54.3301503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3301586Z File "", line 1, in 2022-11-23T03:03:54.3301731Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3301824Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3301960Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3302060Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3302204Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3302266Z self.run() 2022-11-23T03:03:54.3302406Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3302500Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3302756Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3302837Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3303107Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3303218Z getattr(self, test_name)() 2022-11-23T03:03:54.3303492Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3303549Z fn() 2022-11-23T03:03:54.3303824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3303902Z test(self, **param_kwargs) 2022-11-23T03:03:54.3304170Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3304249Z return func(*args, **kwargs) 2022-11-23T03:03:54.3304425Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3304495Z self.run_subtests( 2022-11-23T03:03:54.3304758Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3304869Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3305142Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3305241Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3305522Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3305594Z output = model(*input) 2022-11-23T03:03:54.3305838Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3305924Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3306208Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3306326Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3306606Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3306681Z _lazy_init(state, module) 2022-11-23T03:03:54.3306946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3307037Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3307290Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3307368Z return func(*args, **kwargs) 2022-11-23T03:03:54.3307652Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3307748Z p_assert( 2022-11-23T03:03:54.3307998Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3308076Z traceback.print_stack() 2022-11-23T03:03:54.3308238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3308322Z File "", line 1, in 2022-11-23T03:03:54.3308466Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3308557Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3308695Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3308789Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3308935Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3308996Z self.run() 2022-11-23T03:03:54.3309134Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3309231Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3309484Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3309568Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3309881Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3309960Z getattr(self, test_name)() 2022-11-23T03:03:54.3310231Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3310288Z fn() 2022-11-23T03:03:54.3310563Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3310639Z test(self, **param_kwargs) 2022-11-23T03:03:54.3310910Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3310990Z return func(*args, **kwargs) 2022-11-23T03:03:54.3311163Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3311233Z self.run_subtests( 2022-11-23T03:03:54.3311493Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3311602Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3311874Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3311974Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3312259Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3312334Z output = model(*input) 2022-11-23T03:03:54.3312579Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3312672Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3312952Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3313071Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3313346Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3313421Z _lazy_init(state, module) 2022-11-23T03:03:54.3313683Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3313775Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3314028Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3314107Z return func(*args, **kwargs) 2022-11-23T03:03:54.3314392Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3314485Z p_assert( 2022-11-23T03:03:54.3314732Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3314810Z traceback.print_stack() 2022-11-23T03:03:54.3314975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3315057Z File "", line 1, in 2022-11-23T03:03:54.3315200Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3315292Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3315428Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3315526Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3315672Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3315735Z self.run() 2022-11-23T03:03:54.3315872Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3315968Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3316440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3316624Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3316916Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3316996Z getattr(self, test_name)() 2022-11-23T03:03:54.3317284Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3317339Z fn() 2022-11-23T03:03:54.3317627Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3317707Z test(self, **param_kwargs) 2022-11-23T03:03:54.3317992Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3318074Z return func(*args, **kwargs) 2022-11-23T03:03:54.3318257Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3318333Z self.run_subtests( 2022-11-23T03:03:54.3318612Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3318728Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3319016Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3319121Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3319419Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3319498Z output = model(*input) 2022-11-23T03:03:54.3319754Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3319848Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3320148Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3320273Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3320567Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3320645Z _lazy_init(state, module) 2022-11-23T03:03:54.3320917Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3321015Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3321280Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3321404Z return func(*args, **kwargs) 2022-11-23T03:03:54.3321705Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3321769Z p_assert( 2022-11-23T03:03:54.3322036Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3322120Z traceback.print_stack() 2022-11-23T03:03:54.3322293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3322379Z File "", line 1, in 2022-11-23T03:03:54.3322532Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3322631Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3322777Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3322879Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3323033Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3323098Z self.run() 2022-11-23T03:03:54.3323237Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3323338Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3323639Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3323730Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3324018Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3324099Z getattr(self, test_name)() 2022-11-23T03:03:54.3324384Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3324443Z fn() 2022-11-23T03:03:54.3324731Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3324813Z test(self, **param_kwargs) 2022-11-23T03:03:54.3325101Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3325184Z return func(*args, **kwargs) 2022-11-23T03:03:54.3325369Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3325442Z self.run_subtests( 2022-11-23T03:03:54.3325718Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3325832Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3326123Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3326229Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3326523Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3326604Z output = model(*input) 2022-11-23T03:03:54.3326864Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3326961Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3327358Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3327477Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3327759Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3327834Z _lazy_init(state, module) 2022-11-23T03:03:54.3328097Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3328192Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3328484Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3328565Z return func(*args, **kwargs) 2022-11-23T03:03:54.3328852Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3328915Z p_assert( 2022-11-23T03:03:54.3329166Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3329244Z traceback.print_stack() 2022-11-23T03:03:54.3329409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3329493Z File "", line 1, in 2022-11-23T03:03:54.3329631Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3329724Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3329860Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3329960Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3330105Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3330168Z self.run() 2022-11-23T03:03:54.3330338Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3330433Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3330686Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3330771Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3331044Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3331123Z getattr(self, test_name)() 2022-11-23T03:03:54.3331392Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3331453Z fn() 2022-11-23T03:03:54.3331726Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3331802Z test(self, **param_kwargs) 2022-11-23T03:03:54.3332070Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3332146Z return func(*args, **kwargs) 2022-11-23T03:03:54.3332322Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3332393Z self.run_subtests( 2022-11-23T03:03:54.3332657Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3332766Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3333040Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3333144Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3333427Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3333501Z output = model(*input) 2022-11-23T03:03:54.3333746Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3333837Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3334121Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3334238Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3334512Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3334588Z _lazy_init(state, module) 2022-11-23T03:03:54.3334851Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3334977Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3335232Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3335312Z return func(*args, **kwargs) 2022-11-23T03:03:54.3335594Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3335657Z p_assert( 2022-11-23T03:03:54.3335905Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3335984Z traceback.print_stack() 2022-11-23T03:03:54.3336148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3336231Z File "", line 1, in 2022-11-23T03:03:54.3336373Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3336467Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3336605Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3336703Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3336849Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3336959Z self.run() 2022-11-23T03:03:54.3337098Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3337192Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3337446Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3337531Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3337799Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3337876Z getattr(self, test_name)() 2022-11-23T03:03:54.3338147Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3338206Z fn() 2022-11-23T03:03:54.3338480Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3338557Z test(self, **param_kwargs) 2022-11-23T03:03:54.3338825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3338904Z return func(*args, **kwargs) 2022-11-23T03:03:54.3339076Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3339146Z self.run_subtests( 2022-11-23T03:03:54.3339410Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3339517Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3339790Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3339892Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3340179Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3340254Z output = model(*input) 2022-11-23T03:03:54.3340498Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3340590Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3341000Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3341120Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3341398Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3341978Z _lazy_init(state, module) 2022-11-23T03:03:54.3342477Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3342577Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3342846Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3342932Z return func(*args, **kwargs) 2022-11-23T03:03:54.3343235Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3343297Z p_assert( 2022-11-23T03:03:54.3343566Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3343649Z traceback.print_stack() 2022-11-23T03:03:54.3343822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3343908Z File "", line 1, in 2022-11-23T03:03:54.3344128Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3344324Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3344473Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3344579Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3344769Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3344836Z self.run() 2022-11-23T03:03:54.3344983Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3345084Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3345356Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3345447Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3345737Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3345822Z getattr(self, test_name)() 2022-11-23T03:03:54.3346108Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3346171Z fn() 2022-11-23T03:03:54.3346463Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3346543Z test(self, **param_kwargs) 2022-11-23T03:03:54.3346830Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3346913Z return func(*args, **kwargs) 2022-11-23T03:03:54.3347095Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3347169Z self.run_subtests( 2022-11-23T03:03:54.3347448Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3347562Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3347847Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3347953Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3348253Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3348331Z output = model(*input) 2022-11-23T03:03:54.3348586Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3348681Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3348981Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3349106Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3349401Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3349517Z _lazy_init(state, module) 2022-11-23T03:03:54.3349797Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3349894Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3350166Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3350250Z return func(*args, **kwargs) 2022-11-23T03:03:54.3350556Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3350620Z p_assert( 2022-11-23T03:03:54.3350884Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3350962Z traceback.print_stack() 2022-11-23T03:03:54.3351137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3351229Z File "", line 1, in 2022-11-23T03:03:54.3351380Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3351477Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3351655Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3351760Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3351915Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3351982Z self.run() 2022-11-23T03:03:54.3352129Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3352229Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3352498Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3352587Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3352881Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3352962Z getattr(self, test_name)() 2022-11-23T03:03:54.3353249Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3353309Z fn() 2022-11-23T03:03:54.3353595Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3353677Z test(self, **param_kwargs) 2022-11-23T03:03:54.3353963Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3354048Z return func(*args, **kwargs) 2022-11-23T03:03:54.3354332Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3354403Z self.run_subtests( 2022-11-23T03:03:54.3354667Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3354778Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3355051Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3355151Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3355435Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3355508Z output = model(*input) 2022-11-23T03:03:54.3355753Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3355844Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3356177Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3356468Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3356815Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3356899Z _lazy_init(state, module) 2022-11-23T03:03:54.3357189Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3357286Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3357558Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3357644Z return func(*args, **kwargs) 2022-11-23T03:03:54.3357955Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3358022Z p_assert( 2022-11-23T03:03:54.3358295Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3358383Z traceback.print_stack() 2022-11-23T03:03:54.3358564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3358658Z File "", line 1, in 2022-11-23T03:03:54.3358818Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3358957Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3359109Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3359221Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3359377Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3359445Z self.run() 2022-11-23T03:03:54.3359630Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3359727Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3360005Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3360101Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3360395Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3360478Z getattr(self, test_name)() 2022-11-23T03:03:54.3360778Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3360841Z fn() 2022-11-23T03:03:54.3361138Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3361220Z test(self, **param_kwargs) 2022-11-23T03:03:54.3361507Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3361593Z return func(*args, **kwargs) 2022-11-23T03:03:54.3361783Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3361860Z self.run_subtests( 2022-11-23T03:03:54.3362150Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3362268Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3362572Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3362680Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3362985Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3363060Z output = model(*input) 2022-11-23T03:03:54.3363327Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3363430Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3363738Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3363901Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3364200Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3364283Z _lazy_init(state, module) 2022-11-23T03:03:54.3364572Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3364674Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3364947Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3365033Z return func(*args, **kwargs) 2022-11-23T03:03:54.3365344Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3365411Z p_assert( 2022-11-23T03:03:54.3365686Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3365774Z traceback.print_stack() 2022-11-23T03:03:54.3365952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3366043Z File "", line 1, in 2022-11-23T03:03:54.3366230Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3366327Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3366477Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3366585Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3366743Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3366813Z self.run() 2022-11-23T03:03:54.3367082Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3367289Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3367551Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3367638Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3367910Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3367989Z getattr(self, test_name)() 2022-11-23T03:03:54.3368260Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3368320Z fn() 2022-11-23T03:03:54.3368596Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3368672Z test(self, **param_kwargs) 2022-11-23T03:03:54.3368941Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3369021Z return func(*args, **kwargs) 2022-11-23T03:03:54.3369193Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3369263Z self.run_subtests( 2022-11-23T03:03:54.3369526Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3369639Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3369913Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3370013Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3370299Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3370373Z output = model(*input) 2022-11-23T03:03:54.3370615Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3370706Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3371025Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3371145Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3371426Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3371503Z _lazy_init(state, module) 2022-11-23T03:03:54.3371767Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3371863Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3372120Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3372200Z return func(*args, **kwargs) 2022-11-23T03:03:54.3372484Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3372542Z p_assert( 2022-11-23T03:03:54.3372795Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3372875Z traceback.print_stack() 2022-11-23T03:03:54.3373040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3373153Z File "", line 1, in 2022-11-23T03:03:54.3373297Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3373389Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3373530Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3373628Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3373771Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3373834Z self.run() 2022-11-23T03:03:54.3373971Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3374069Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3374324Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3374409Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3374683Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3374756Z getattr(self, test_name)() 2022-11-23T03:03:54.3375027Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3375085Z fn() 2022-11-23T03:03:54.3375360Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3375436Z test(self, **param_kwargs) 2022-11-23T03:03:54.3375705Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3375786Z return func(*args, **kwargs) 2022-11-23T03:03:54.3375959Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3376029Z self.run_subtests( 2022-11-23T03:03:54.3376297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3376405Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3376677Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3376778Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3377063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3377139Z output = model(*input) 2022-11-23T03:03:54.3377381Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3377504Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3377789Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3377910Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3378184Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3378259Z _lazy_init(state, module) 2022-11-23T03:03:54.3378522Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3378614Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3378867Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3378947Z return func(*args, **kwargs) 2022-11-23T03:03:54.3379232Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3379293Z p_assert( 2022-11-23T03:03:54.3379543Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3379621Z traceback.print_stack() 2022-11-23T03:03:54.3379818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3379902Z File "", line 1, in 2022-11-23T03:03:54.3380047Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3380138Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3380274Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3380373Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3380520Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3380579Z self.run() 2022-11-23T03:03:54.3380720Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3380816Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3381069Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3381157Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3381428Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3381506Z getattr(self, test_name)() 2022-11-23T03:03:54.3381776Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3381835Z fn() 2022-11-23T03:03:54.3382109Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3382186Z test(self, **param_kwargs) 2022-11-23T03:03:54.3382454Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3382533Z return func(*args, **kwargs) 2022-11-23T03:03:54.3382707Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3382778Z self.run_subtests( 2022-11-23T03:03:54.3383041Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3383150Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3383419Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3383520Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3383804Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3383913Z output = model(*input) 2022-11-23T03:03:54.3384159Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3384253Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3384538Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3384656Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3384933Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3385008Z _lazy_init(state, module) 2022-11-23T03:03:54.3385273Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3385367Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3385623Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3385705Z return func(*args, **kwargs) 2022-11-23T03:03:54.3385990Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3386051Z p_assert( 2022-11-23T03:03:54.3386333Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3386416Z traceback.print_stack() 2022-11-23T03:03:54.3386575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3386659Z File "", line 1, in 2022-11-23T03:03:54.3386801Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3386893Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3387030Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3387129Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3387275Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3387338Z self.run() 2022-11-23T03:03:54.3387472Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3387568Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3387825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3387911Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3388182Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3388259Z getattr(self, test_name)() 2022-11-23T03:03:54.3388529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3388587Z fn() 2022-11-23T03:03:54.3388864Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3388938Z test(self, **param_kwargs) 2022-11-23T03:03:54.3389204Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3389284Z return func(*args, **kwargs) 2022-11-23T03:03:54.3389460Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3389532Z self.run_subtests( 2022-11-23T03:03:54.3389794Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3389904Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3390177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3390278Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3390560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3390680Z output = model(*input) 2022-11-23T03:03:54.3390928Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3391019Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3391302Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3391422Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3391700Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3391778Z _lazy_init(state, module) 2022-11-23T03:03:54.3392041Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3392134Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3392385Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3392464Z return func(*args, **kwargs) 2022-11-23T03:03:54.3392754Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3392847Z p_assert( 2022-11-23T03:03:54.3393099Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3393177Z traceback.print_stack() 2022-11-23T03:03:54.3393341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3393423Z File "", line 1, in 2022-11-23T03:03:54.3393565Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3393657Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3393795Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3393897Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3394043Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3394106Z self.run() 2022-11-23T03:03:54.3394246Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3394341Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3394588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3394673Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3394945Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3395021Z getattr(self, test_name)() 2022-11-23T03:03:54.3395291Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3395350Z fn() 2022-11-23T03:03:54.3395624Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3395703Z test(self, **param_kwargs) 2022-11-23T03:03:54.3395974Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3396103Z return func(*args, **kwargs) 2022-11-23T03:03:54.3396448Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3396524Z self.run_subtests( 2022-11-23T03:03:54.3396803Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3396917Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3397212Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3397362Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3397660Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3397739Z output = model(*input) 2022-11-23T03:03:54.3397992Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3398090Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3398393Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3398517Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3398808Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3398889Z _lazy_init(state, module) 2022-11-23T03:03:54.3399172Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3399270Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3399534Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3399616Z return func(*args, **kwargs) 2022-11-23T03:03:54.3399958Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3400023Z p_assert( 2022-11-23T03:03:54.3400287Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3400371Z traceback.print_stack() 2022-11-23T03:03:54.3400544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3400630Z File "", line 1, in 2022-11-23T03:03:54.3400778Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3400878Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3401018Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3401123Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3401279Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3401347Z self.run() 2022-11-23T03:03:54.3401493Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3401593Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3401861Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3401951Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3402242Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3402325Z getattr(self, test_name)() 2022-11-23T03:03:54.3402611Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3402673Z fn() 2022-11-23T03:03:54.3402964Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3403045Z test(self, **param_kwargs) 2022-11-23T03:03:54.3403328Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3403411Z return func(*args, **kwargs) 2022-11-23T03:03:54.3403594Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3403663Z self.run_subtests( 2022-11-23T03:03:54.3403942Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3404055Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3404466Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3404571Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3404867Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3404946Z output = model(*input) 2022-11-23T03:03:54.3405205Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3405300Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3405598Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3405724Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3406017Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3406099Z _lazy_init(state, module) 2022-11-23T03:03:54.3406377Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3406474Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3406772Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3406857Z return func(*args, **kwargs) 2022-11-23T03:03:54.3407160Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3407333Z p_assert( 2022-11-23T03:03:54.3407584Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3407665Z traceback.print_stack() 2022-11-23T03:03:54.3407830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3407915Z File "", line 1, in 2022-11-23T03:03:54.3408063Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3408156Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3408294Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3408394Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3408541Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3408603Z self.run() 2022-11-23T03:03:54.3408742Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3408838Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3409091Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3409177Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3409452Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3409536Z getattr(self, test_name)() 2022-11-23T03:03:54.3409804Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3409861Z fn() 2022-11-23T03:03:54.3410138Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3410216Z test(self, **param_kwargs) 2022-11-23T03:03:54.3410484Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3410563Z return func(*args, **kwargs) 2022-11-23T03:03:54.3410740Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3410809Z self.run_subtests( 2022-11-23T03:03:54.3411074Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3411216Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3411492Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3411592Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3411876Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3411951Z output = model(*input) 2022-11-23T03:03:54.3412195Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3412285Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3412571Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3412689Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3412966Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3413039Z _lazy_init(state, module) 2022-11-23T03:03:54.3413303Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3413395Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3413680Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3413761Z return func(*args, **kwargs) 2022-11-23T03:03:54.3414046Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3414108Z p_assert( 2022-11-23T03:03:54.3414357Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3414436Z traceback.print_stack() 2022-11-23T03:03:54.3414604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3414771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3414934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3415096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3415254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3415411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3415567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3415723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3415880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3416030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3416189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3416344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3416501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3416657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3416813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3416965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3417119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3417274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3417461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3417616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3417768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3417926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3418081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3418234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3418387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3418541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3418694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3418851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3419003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3419158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3419336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3419492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3419680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3419869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3420057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3420243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3420434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3420623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3420814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3421003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3425515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3425714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3425880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3426044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3426205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3426364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3426523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3426593Z dist init r=0, world=2 2022-11-23T03:03:54.3426835Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3427068Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3427305Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3427532Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3427870Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3428103Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3428326Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3428539Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3428765Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3428987Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3429243Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3429470Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3429537Z dist init r=1, world=2 2022-11-23T03:03:54.3429756Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3429969Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3430193Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3430414Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3430629Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3430858Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3431081Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3431300Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3431525Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3431751Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3431968Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3432190Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3432253Z ok (53.244s) 2022-11-23T03:03:54.3432408Z test_delayed_reduce_scatter_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T03:03:54.3432664Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13572 2022-11-23T03:03:54.3432816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13573 2022-11-23T03:03:54.3433140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.3433259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.3433552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.3433683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.3433857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:03:54.3434132Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:03:54.3434252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:03:54.3434538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:03:54.3434701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:03:54.3434875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:03:54.3435179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.3435479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:03:54.3435634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:03:54.3435791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:03:54.3435956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3436167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3437213Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.3437288Z warnings.warn( 2022-11-23T03:03:54.3437379Z File "", line 1, in 2022-11-23T03:03:54.3437538Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3437642Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3437795Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3437902Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3438063Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3438132Z self.run() 2022-11-23T03:03:54.3438282Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3438387Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3438666Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3438758Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3439057Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3439140Z getattr(self, test_name)() 2022-11-23T03:03:54.3439481Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3439539Z fn() 2022-11-23T03:03:54.3439838Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3439921Z test(self, **param_kwargs) 2022-11-23T03:03:54.3440214Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3440302Z return func(*args, **kwargs) 2022-11-23T03:03:54.3440496Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3440571Z self.run_subtests( 2022-11-23T03:03:54.3440862Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3440980Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3441280Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3441389Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3441696Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3441813Z output = model(*input) 2022-11-23T03:03:54.3442083Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3442183Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3442493Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3442622Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3442925Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3443009Z _lazy_init(state, module) 2022-11-23T03:03:54.3443292Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3443393Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3443668Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3443753Z return func(*args, **kwargs) 2022-11-23T03:03:54.3444065Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3444130Z p_assert( 2022-11-23T03:03:54.3444402Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3444489Z traceback.print_stack() 2022-11-23T03:03:54.3444668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3445546Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:03:54.3445623Z warnings.warn( 2022-11-23T03:03:54.3445715Z File "", line 1, in 2022-11-23T03:03:54.3445874Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3445975Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3446127Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3446233Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3446391Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3446495Z self.run() 2022-11-23T03:03:54.3446646Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3446745Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3447246Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3447333Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3447608Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3447686Z getattr(self, test_name)() 2022-11-23T03:03:54.3447956Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3448015Z fn() 2022-11-23T03:03:54.3448289Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3448365Z test(self, **param_kwargs) 2022-11-23T03:03:54.3448633Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3448711Z return func(*args, **kwargs) 2022-11-23T03:03:54.3448885Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3448986Z self.run_subtests( 2022-11-23T03:03:54.3449252Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3449360Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3449635Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3449734Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3450014Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3450086Z output = model(*input) 2022-11-23T03:03:54.3450331Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3450423Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3450706Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3450825Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3451102Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3451179Z _lazy_init(state, module) 2022-11-23T03:03:54.3451443Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3451537Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3451790Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3451872Z return func(*args, **kwargs) 2022-11-23T03:03:54.3452157Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3452217Z p_assert( 2022-11-23T03:03:54.3452470Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3452549Z traceback.print_stack() 2022-11-23T03:03:54.3452714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3452797Z File "", line 1, in 2022-11-23T03:03:54.3452942Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3453030Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3453166Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3453264Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3453445Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3453508Z self.run() 2022-11-23T03:03:54.3453647Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3453741Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3453997Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3454083Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3454355Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3454431Z getattr(self, test_name)() 2022-11-23T03:03:54.3454704Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3454763Z fn() 2022-11-23T03:03:54.3455037Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3455115Z test(self, **param_kwargs) 2022-11-23T03:03:54.3455381Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3455460Z return func(*args, **kwargs) 2022-11-23T03:03:54.3455661Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3455732Z self.run_subtests( 2022-11-23T03:03:54.3455995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3456104Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3456377Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3456478Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3456762Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3456840Z output = model(*input) 2022-11-23T03:03:54.3457083Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3457174Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3457460Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3457580Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3457857Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3457934Z _lazy_init(state, module) 2022-11-23T03:03:54.3458200Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3458294Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3458554Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3458631Z return func(*args, **kwargs) 2022-11-23T03:03:54.3458918Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3458977Z p_assert( 2022-11-23T03:03:54.3459228Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3459307Z traceback.print_stack() 2022-11-23T03:03:54.3459470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3459554Z File "", line 1, in 2022-11-23T03:03:54.3459696Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3459789Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3459924Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3460058Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3460204Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3460265Z self.run() 2022-11-23T03:03:54.3460402Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3460495Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3460748Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3460833Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3461106Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3461178Z getattr(self, test_name)() 2022-11-23T03:03:54.3461450Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3461512Z fn() 2022-11-23T03:03:54.3461785Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3461860Z test(self, **param_kwargs) 2022-11-23T03:03:54.3462126Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3462240Z return func(*args, **kwargs) 2022-11-23T03:03:54.3462419Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3462488Z self.run_subtests( 2022-11-23T03:03:54.3462752Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3462859Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3463133Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3463234Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3463518Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3463592Z output = model(*input) 2022-11-23T03:03:54.3463838Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3463929Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3464213Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3464327Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3464601Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3464677Z _lazy_init(state, module) 2022-11-23T03:03:54.3464937Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3465031Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3465284Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3465361Z return func(*args, **kwargs) 2022-11-23T03:03:54.3465649Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3465710Z p_assert( 2022-11-23T03:03:54.3465960Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3466039Z traceback.print_stack() 2022-11-23T03:03:54.3466203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3466288Z File "", line 1, in 2022-11-23T03:03:54.3466432Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3466559Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3466695Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3466795Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3466941Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3467000Z self.run() 2022-11-23T03:03:54.3467138Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3467232Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3467487Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3467571Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3467843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3467919Z getattr(self, test_name)() 2022-11-23T03:03:54.3468190Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3468252Z fn() 2022-11-23T03:03:54.3468527Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3468603Z test(self, **param_kwargs) 2022-11-23T03:03:54.3468903Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3468983Z return func(*args, **kwargs) 2022-11-23T03:03:54.3469157Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3469227Z self.run_subtests( 2022-11-23T03:03:54.3469494Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3469602Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3469870Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3469974Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3470258Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3470336Z output = model(*input) 2022-11-23T03:03:54.3470577Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3470671Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3470955Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3471074Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3471348Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3471423Z _lazy_init(state, module) 2022-11-23T03:03:54.3471689Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3471782Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3472033Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3472113Z return func(*args, **kwargs) 2022-11-23T03:03:54.3472400Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3472460Z p_assert( 2022-11-23T03:03:54.3472713Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3472792Z traceback.print_stack() 2022-11-23T03:03:54.3472952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3473035Z File "", line 1, in 2022-11-23T03:03:54.3473231Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3473327Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3473465Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3473563Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3473710Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3473774Z self.run() 2022-11-23T03:03:54.3473910Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3474007Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3474260Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3474346Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3474619Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3474699Z getattr(self, test_name)() 2022-11-23T03:03:54.3474969Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3475027Z fn() 2022-11-23T03:03:54.3475331Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3475404Z test(self, **param_kwargs) 2022-11-23T03:03:54.3475671Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3475750Z return func(*args, **kwargs) 2022-11-23T03:03:54.3475924Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3475994Z self.run_subtests( 2022-11-23T03:03:54.3476594Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3476712Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3477001Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3477106Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3477407Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3477486Z output = model(*input) 2022-11-23T03:03:54.3477744Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3477839Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3478137Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3478262Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3478557Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3478640Z _lazy_init(state, module) 2022-11-23T03:03:54.3478918Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3479016Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3479281Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3479363Z return func(*args, **kwargs) 2022-11-23T03:03:54.3479665Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3479730Z p_assert( 2022-11-23T03:03:54.3479998Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3480083Z traceback.print_stack() 2022-11-23T03:03:54.3480258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3480390Z File "", line 1, in 2022-11-23T03:03:54.3480542Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3480639Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3480788Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3480898Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3481055Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3481122Z self.run() 2022-11-23T03:03:54.3481272Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3481372Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3481638Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3481729Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3482022Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3482108Z getattr(self, test_name)() 2022-11-23T03:03:54.3482397Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3482459Z fn() 2022-11-23T03:03:54.3482786Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3482874Z test(self, **param_kwargs) 2022-11-23T03:03:54.3483157Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3483242Z return func(*args, **kwargs) 2022-11-23T03:03:54.3483427Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3483501Z self.run_subtests( 2022-11-23T03:03:54.3483778Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3483894Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3484184Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3484292Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3484592Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3484672Z output = model(*input) 2022-11-23T03:03:54.3484924Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3485020Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3485319Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3485444Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3485740Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3485823Z _lazy_init(state, module) 2022-11-23T03:03:54.3486100Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3486201Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3486469Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3486552Z return func(*args, **kwargs) 2022-11-23T03:03:54.3486856Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3486920Z p_assert( 2022-11-23T03:03:54.3487284Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3487395Z traceback.print_stack() 2022-11-23T03:03:54.3487560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3487643Z File "", line 1, in 2022-11-23T03:03:54.3487790Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3487886Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3488018Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3488119Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3488263Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3488327Z self.run() 2022-11-23T03:03:54.3488468Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3488563Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3488816Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3488903Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3489176Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3489252Z getattr(self, test_name)() 2022-11-23T03:03:54.3489553Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3489611Z fn() 2022-11-23T03:03:54.3489887Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3489964Z test(self, **param_kwargs) 2022-11-23T03:03:54.3490230Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3490308Z return func(*args, **kwargs) 2022-11-23T03:03:54.3490481Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3490548Z self.run_subtests( 2022-11-23T03:03:54.3490809Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3490919Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3491194Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3491295Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3491581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3491656Z output = model(*input) 2022-11-23T03:03:54.3491900Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3491989Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3492272Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3492391Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3492668Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3492744Z _lazy_init(state, module) 2022-11-23T03:03:54.3493010Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3493102Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3493354Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3493431Z return func(*args, **kwargs) 2022-11-23T03:03:54.3493720Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3493780Z p_assert( 2022-11-23T03:03:54.3494027Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3494146Z traceback.print_stack() 2022-11-23T03:03:54.3494310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3494392Z File "", line 1, in 2022-11-23T03:03:54.3494535Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3494628Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3494767Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3494865Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3495010Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3495073Z self.run() 2022-11-23T03:03:54.3495211Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3495305Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3495565Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3495649Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3495921Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3496030Z getattr(self, test_name)() 2022-11-23T03:03:54.3496298Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3496354Z fn() 2022-11-23T03:03:54.3496630Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3496705Z test(self, **param_kwargs) 2022-11-23T03:03:54.3496968Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3497045Z return func(*args, **kwargs) 2022-11-23T03:03:54.3497221Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3497290Z self.run_subtests( 2022-11-23T03:03:54.3497553Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3497665Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3497937Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3498036Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3498318Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3498391Z output = model(*input) 2022-11-23T03:03:54.3498635Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3498725Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3499011Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3499129Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3499407Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3499478Z _lazy_init(state, module) 2022-11-23T03:03:54.3499742Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3499835Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3500085Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3500164Z return func(*args, **kwargs) 2022-11-23T03:03:54.3500450Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3500546Z p_assert( 2022-11-23T03:03:54.3500800Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3500878Z traceback.print_stack() 2022-11-23T03:03:54.3501041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3501125Z File "", line 1, in 2022-11-23T03:03:54.3501270Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3501361Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3501498Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3501596Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3501738Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3501800Z self.run() 2022-11-23T03:03:54.3501933Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3502029Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3502282Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3502366Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3502668Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3502746Z getattr(self, test_name)() 2022-11-23T03:03:54.3503017Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3503074Z fn() 2022-11-23T03:03:54.3503349Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3503424Z test(self, **param_kwargs) 2022-11-23T03:03:54.3503694Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3503774Z return func(*args, **kwargs) 2022-11-23T03:03:54.3503948Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3504016Z self.run_subtests( 2022-11-23T03:03:54.3504281Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3504389Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3504663Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3504758Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3505037Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3505110Z output = model(*input) 2022-11-23T03:03:54.3505351Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3505444Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3505723Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3505843Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3506117Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3506193Z _lazy_init(state, module) 2022-11-23T03:03:54.3506456Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3506549Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3506804Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3506880Z return func(*args, **kwargs) 2022-11-23T03:03:54.3507201Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3507260Z p_assert( 2022-11-23T03:03:54.3507511Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3507589Z traceback.print_stack() 2022-11-23T03:03:54.3507756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3507834Z File "", line 1, in 2022-11-23T03:03:54.3507976Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3508067Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3508203Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3508302Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3508446Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3508512Z self.run() 2022-11-23T03:03:54.3508650Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3508744Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3508994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3509108Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3509382Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3509460Z getattr(self, test_name)() 2022-11-23T03:03:54.3509727Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3509784Z fn() 2022-11-23T03:03:54.3510058Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3510133Z test(self, **param_kwargs) 2022-11-23T03:03:54.3510399Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3510476Z return func(*args, **kwargs) 2022-11-23T03:03:54.3510648Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3510718Z self.run_subtests( 2022-11-23T03:03:54.3510980Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3511087Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3511361Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3511459Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3511743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3511818Z output = model(*input) 2022-11-23T03:03:54.3512060Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3512150Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3512434Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3512552Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3512823Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3512897Z _lazy_init(state, module) 2022-11-23T03:03:54.3513160Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3513253Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3513504Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3513610Z return func(*args, **kwargs) 2022-11-23T03:03:54.3513896Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3513956Z p_assert( 2022-11-23T03:03:54.3514210Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3514288Z traceback.print_stack() 2022-11-23T03:03:54.3514451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3514532Z File "", line 1, in 2022-11-23T03:03:54.3514675Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3514768Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3514904Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3515002Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3515148Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3515210Z self.run() 2022-11-23T03:03:54.3515343Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3515439Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3515722Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3515804Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3516137Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3516213Z getattr(self, test_name)() 2022-11-23T03:03:54.3516484Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3516543Z fn() 2022-11-23T03:03:54.3516812Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3516891Z test(self, **param_kwargs) 2022-11-23T03:03:54.3517155Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3517232Z return func(*args, **kwargs) 2022-11-23T03:03:54.3517408Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3517477Z self.run_subtests( 2022-11-23T03:03:54.3517739Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3517845Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3518118Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3518217Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3518496Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3518571Z output = model(*input) 2022-11-23T03:03:54.3518814Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3518900Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3519182Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3519298Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3519571Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3519646Z _lazy_init(state, module) 2022-11-23T03:03:54.3519911Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3520003Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3520297Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3520375Z return func(*args, **kwargs) 2022-11-23T03:03:54.3520659Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3520721Z p_assert( 2022-11-23T03:03:54.3520974Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3521052Z traceback.print_stack() 2022-11-23T03:03:54.3521215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3521297Z File "", line 1, in 2022-11-23T03:03:54.3521441Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3521533Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3521667Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3521764Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3521908Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3521971Z self.run() 2022-11-23T03:03:54.3522158Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3522255Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3522507Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3522593Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3522863Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3522941Z getattr(self, test_name)() 2022-11-23T03:03:54.3523211Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3523270Z fn() 2022-11-23T03:03:54.3523542Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3523617Z test(self, **param_kwargs) 2022-11-23T03:03:54.3523885Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3523964Z return func(*args, **kwargs) 2022-11-23T03:03:54.3524137Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3524206Z self.run_subtests( 2022-11-23T03:03:54.3524464Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3524570Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3524841Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3524945Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3525225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3525299Z output = model(*input) 2022-11-23T03:03:54.3525543Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3525633Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3525915Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3526032Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3526307Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3526382Z _lazy_init(state, module) 2022-11-23T03:03:54.3526645Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3526771Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3527024Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3527103Z return func(*args, **kwargs) 2022-11-23T03:03:54.3527390Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3527450Z p_assert( 2022-11-23T03:03:54.3527699Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3527773Z traceback.print_stack() 2022-11-23T03:03:54.3527936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3528018Z File "", line 1, in 2022-11-23T03:03:54.3528159Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3528252Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3528389Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3528486Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3528633Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3528728Z self.run() 2022-11-23T03:03:54.3528867Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3528963Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3529216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3529299Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3529570Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3529645Z getattr(self, test_name)() 2022-11-23T03:03:54.3529916Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3529968Z fn() 2022-11-23T03:03:54.3530242Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3530317Z test(self, **param_kwargs) 2022-11-23T03:03:54.3530586Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3530664Z return func(*args, **kwargs) 2022-11-23T03:03:54.3530835Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3530905Z self.run_subtests( 2022-11-23T03:03:54.3531166Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3531273Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3531546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3531644Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3531926Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3532001Z output = model(*input) 2022-11-23T03:03:54.3532244Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3532334Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3532614Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3532731Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3533004Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3533115Z _lazy_init(state, module) 2022-11-23T03:03:54.3533372Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3533465Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3533718Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3533796Z return func(*args, **kwargs) 2022-11-23T03:03:54.3534080Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3534140Z p_assert( 2022-11-23T03:03:54.3534389Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3534466Z traceback.print_stack() 2022-11-23T03:03:54.3534629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3534710Z File "", line 1, in 2022-11-23T03:03:54.3534853Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3534944Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3535081Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3535178Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3535358Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3535419Z self.run() 2022-11-23T03:03:54.3535553Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3535645Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3535899Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3535982Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3536252Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3536329Z getattr(self, test_name)() 2022-11-23T03:03:54.3536598Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3536656Z fn() 2022-11-23T03:03:54.3536934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3537011Z test(self, **param_kwargs) 2022-11-23T03:03:54.3537276Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3537353Z return func(*args, **kwargs) 2022-11-23T03:03:54.3537527Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3537596Z self.run_subtests( 2022-11-23T03:03:54.3537860Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3537969Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3538241Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3538340Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3538618Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3538692Z output = model(*input) 2022-11-23T03:03:54.3538936Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3539025Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3539305Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3539421Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3539698Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3539805Z _lazy_init(state, module) 2022-11-23T03:03:54.3540070Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3540162Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3540416Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3540495Z return func(*args, **kwargs) 2022-11-23T03:03:54.3540778Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3540839Z p_assert( 2022-11-23T03:03:54.3541089Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3541167Z traceback.print_stack() 2022-11-23T03:03:54.3541331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3541415Z File "", line 1, in 2022-11-23T03:03:54.3541553Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3541645Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3541813Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3541913Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3542059Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3542120Z self.run() 2022-11-23T03:03:54.3542257Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3542350Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3542601Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3542685Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3542958Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3543034Z getattr(self, test_name)() 2022-11-23T03:03:54.3543306Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3543365Z fn() 2022-11-23T03:03:54.3543637Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3543712Z test(self, **param_kwargs) 2022-11-23T03:03:54.3543978Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3544052Z return func(*args, **kwargs) 2022-11-23T03:03:54.3544224Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3544292Z self.run_subtests( 2022-11-23T03:03:54.3544557Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3544665Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3544939Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3545040Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3545319Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3545392Z output = model(*input) 2022-11-23T03:03:54.3545636Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3545728Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3546011Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3546162Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3546442Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3546518Z _lazy_init(state, module) 2022-11-23T03:03:54.3546786Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3546879Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3547133Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3547210Z return func(*args, **kwargs) 2022-11-23T03:03:54.3547488Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3547548Z p_assert( 2022-11-23T03:03:54.3547798Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3547879Z traceback.print_stack() 2022-11-23T03:03:54.3548042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3548125Z File "", line 1, in 2022-11-23T03:03:54.3548270Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3548394Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3548531Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3548629Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3548773Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3548834Z self.run() 2022-11-23T03:03:54.3548975Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3549069Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3549322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3549409Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3549673Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3549748Z getattr(self, test_name)() 2022-11-23T03:03:54.3550016Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3550073Z fn() 2022-11-23T03:03:54.3550342Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3550418Z test(self, **param_kwargs) 2022-11-23T03:03:54.3550683Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3550763Z return func(*args, **kwargs) 2022-11-23T03:03:54.3550937Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3551009Z self.run_subtests( 2022-11-23T03:03:54.3551272Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3551380Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3551652Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3551753Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3552033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3552106Z output = model(*input) 2022-11-23T03:03:54.3552348Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3552438Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3552716Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3552866Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3553143Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3553220Z _lazy_init(state, module) 2022-11-23T03:03:54.3553483Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3553575Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3553831Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3553909Z return func(*args, **kwargs) 2022-11-23T03:03:54.3554194Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3554255Z p_assert( 2022-11-23T03:03:54.3554506Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3554584Z traceback.print_stack() 2022-11-23T03:03:54.3554748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3554830Z File "", line 1, in 2022-11-23T03:03:54.3555021Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3555114Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3555251Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3555348Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3555487Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3555549Z self.run() 2022-11-23T03:03:54.3555685Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3555780Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3556035Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3556165Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3556440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3556520Z getattr(self, test_name)() 2022-11-23T03:03:54.3556789Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3556847Z fn() 2022-11-23T03:03:54.3557122Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3557198Z test(self, **param_kwargs) 2022-11-23T03:03:54.3557467Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3557546Z return func(*args, **kwargs) 2022-11-23T03:03:54.3557724Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3557792Z self.run_subtests( 2022-11-23T03:03:54.3558055Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3558161Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3558433Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3558531Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3558814Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3558887Z output = model(*input) 2022-11-23T03:03:54.3559130Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3559265Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3559551Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3559667Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3559946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3560021Z _lazy_init(state, module) 2022-11-23T03:03:54.3560288Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3560382Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3560632Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3560710Z return func(*args, **kwargs) 2022-11-23T03:03:54.3560995Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3561057Z p_assert( 2022-11-23T03:03:54.3561307Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3561387Z traceback.print_stack() 2022-11-23T03:03:54.3561582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3561665Z File "", line 1, in 2022-11-23T03:03:54.3561807Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3561899Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3562036Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3562135Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3562281Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3562344Z self.run() 2022-11-23T03:03:54.3562481Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3562578Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3562832Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3562917Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3563194Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3563270Z getattr(self, test_name)() 2022-11-23T03:03:54.3563539Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3563596Z fn() 2022-11-23T03:03:54.3563866Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3563941Z test(self, **param_kwargs) 2022-11-23T03:03:54.3564208Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3564288Z return func(*args, **kwargs) 2022-11-23T03:03:54.3564460Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3564529Z self.run_subtests( 2022-11-23T03:03:54.3564793Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3564899Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3565170Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3565270Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3565552Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3565625Z output = model(*input) 2022-11-23T03:03:54.3565868Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3565991Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3566276Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3566395Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3566671Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3566746Z _lazy_init(state, module) 2022-11-23T03:03:54.3567008Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3567097Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3567349Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3567427Z return func(*args, **kwargs) 2022-11-23T03:03:54.3567711Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3567771Z p_assert( 2022-11-23T03:03:54.3568025Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3568138Z traceback.print_stack() 2022-11-23T03:03:54.3568305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3568387Z File "", line 1, in 2022-11-23T03:03:54.3568528Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3568620Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3568756Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3568854Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3568997Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3569060Z self.run() 2022-11-23T03:03:54.3569197Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3569287Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3569544Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3569629Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3569900Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3569976Z getattr(self, test_name)() 2022-11-23T03:03:54.3570245Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3570302Z fn() 2022-11-23T03:03:54.3570575Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3570653Z test(self, **param_kwargs) 2022-11-23T03:03:54.3570918Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3570995Z return func(*args, **kwargs) 2022-11-23T03:03:54.3571170Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3571239Z self.run_subtests( 2022-11-23T03:03:54.3571501Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3571608Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3571881Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3571978Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3572259Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3572383Z output = model(*input) 2022-11-23T03:03:54.3572624Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3572716Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3573000Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3573118Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3573396Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3573470Z _lazy_init(state, module) 2022-11-23T03:03:54.3573734Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3573827Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3574077Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3574156Z return func(*args, **kwargs) 2022-11-23T03:03:54.3574437Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3574498Z p_assert( 2022-11-23T03:03:54.3574785Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3574865Z traceback.print_stack() 2022-11-23T03:03:54.3575028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3575110Z File "", line 1, in 2022-11-23T03:03:54.3575252Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3575339Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3575474Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3575571Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3575717Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3575779Z self.run() 2022-11-23T03:03:54.3575917Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3576014Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3576267Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3576350Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3576623Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3576699Z getattr(self, test_name)() 2022-11-23T03:03:54.3576967Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3577023Z fn() 2022-11-23T03:03:54.3577297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3577374Z test(self, **param_kwargs) 2022-11-23T03:03:54.3577645Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3577723Z return func(*args, **kwargs) 2022-11-23T03:03:54.3577892Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3577961Z self.run_subtests( 2022-11-23T03:03:54.3578225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3578333Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3578604Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3578702Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3579020Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3579093Z output = model(*input) 2022-11-23T03:03:54.3579336Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3579429Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3579712Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3579828Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3580104Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3580179Z _lazy_init(state, module) 2022-11-23T03:03:54.3580446Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3580541Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3580795Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3580874Z return func(*args, **kwargs) 2022-11-23T03:03:54.3581191Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3581248Z p_assert( 2022-11-23T03:03:54.3581498Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3581576Z traceback.print_stack() 2022-11-23T03:03:54.3581739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3581821Z File "", line 1, in 2022-11-23T03:03:54.3581962Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3582054Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3582195Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3582293Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3582437Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3582499Z self.run() 2022-11-23T03:03:54.3582635Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3582729Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3582980Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3583064Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3583337Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3583408Z getattr(self, test_name)() 2022-11-23T03:03:54.3583676Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3583736Z fn() 2022-11-23T03:03:54.3584007Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3584082Z test(self, **param_kwargs) 2022-11-23T03:03:54.3584349Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3584426Z return func(*args, **kwargs) 2022-11-23T03:03:54.3584597Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3584665Z self.run_subtests( 2022-11-23T03:03:54.3584930Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3585036Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3585312Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3585446Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3585727Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3585800Z output = model(*input) 2022-11-23T03:03:54.3586047Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3586138Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3586420Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3586537Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3586809Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3586883Z _lazy_init(state, module) 2022-11-23T03:03:54.3587147Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3587240Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3587494Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3587572Z return func(*args, **kwargs) 2022-11-23T03:03:54.3587887Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3587949Z p_assert( 2022-11-23T03:03:54.3588201Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3588280Z traceback.print_stack() 2022-11-23T03:03:54.3588444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3588525Z File "", line 1, in 2022-11-23T03:03:54.3588668Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3588762Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3588898Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3588996Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3589143Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3589201Z self.run() 2022-11-23T03:03:54.3589337Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3589430Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3589682Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3589765Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3590039Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3590117Z getattr(self, test_name)() 2022-11-23T03:03:54.3590392Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3590450Z fn() 2022-11-23T03:03:54.3590729Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3590807Z test(self, **param_kwargs) 2022-11-23T03:03:54.3591071Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3591150Z return func(*args, **kwargs) 2022-11-23T03:03:54.3591323Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3591392Z self.run_subtests( 2022-11-23T03:03:54.3591653Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3591760Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3592063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3592163Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3592442Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3592518Z output = model(*input) 2022-11-23T03:03:54.3592759Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3592850Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3593131Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3593247Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3593524Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3593601Z _lazy_init(state, module) 2022-11-23T03:03:54.3593861Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3593952Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3594239Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3594317Z return func(*args, **kwargs) 2022-11-23T03:03:54.3594604Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3594664Z p_assert( 2022-11-23T03:03:54.3594914Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3594993Z traceback.print_stack() 2022-11-23T03:03:54.3595157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3595235Z File "", line 1, in 2022-11-23T03:03:54.3595379Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:03:54.3595471Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:03:54.3595607Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:03:54.3595708Z return self._bootstrap(parent_sentinel) 2022-11-23T03:03:54.3595851Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:03:54.3595914Z self.run() 2022-11-23T03:03:54.3596103Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:03:54.3596198Z self._target(*self._args, **self._kwargs) 2022-11-23T03:03:54.3596453Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:03:54.3596537Z self.run_test(test_name, pipe) 2022-11-23T03:03:54.3596811Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:03:54.3596888Z getattr(self, test_name)() 2022-11-23T03:03:54.3597157Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:03:54.3597214Z fn() 2022-11-23T03:03:54.3597491Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:03:54.3597563Z test(self, **param_kwargs) 2022-11-23T03:03:54.3597831Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:03:54.3597909Z return func(*args, **kwargs) 2022-11-23T03:03:54.3598081Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:03:54.3598149Z self.run_subtests( 2022-11-23T03:03:54.3598410Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:03:54.3598559Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:03:54.3598833Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:03:54.3598933Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:03:54.3599217Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:03:54.3599291Z output = model(*input) 2022-11-23T03:03:54.3599533Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:03:54.3599624Z return forward_call(*input, **kwargs) 2022-11-23T03:03:54.3599907Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:03:54.3600026Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:03:54.3600304Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:03:54.3600380Z _lazy_init(state, module) 2022-11-23T03:03:54.3600647Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:03:54.3600774Z handle.init_flat_param_attributes() 2022-11-23T03:03:54.3601024Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:03:54.3601103Z return func(*args, **kwargs) 2022-11-23T03:03:54.3601387Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:03:54.3601447Z p_assert( 2022-11-23T03:03:54.3601699Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:03:54.3601777Z traceback.print_stack() 2022-11-23T03:03:54.3601940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3602105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3602260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3602423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3602583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3602741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3602896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3603048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3603206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3603362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3603519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3603669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3603822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3603974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3604131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3604284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3604438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3604593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3604778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3604932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3605086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3605245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3605397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3605549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3605703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3605855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3606009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3606162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3606314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3606498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3606652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3606807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3606959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3607108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3607260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3607414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3607570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3607721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3607874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3608027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3608181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3608333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3608485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3608636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3608790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3608943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3609096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:03:54.3609166Z dist init r=1, world=2 2022-11-23T03:03:54.3609408Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3609637Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3609860Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3610093Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3610354Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3610575Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3610807Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3611030Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3611247Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3611480Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3611733Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3611953Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:03:54.3612021Z dist init r=0, world=2 2022-11-23T03:03:54.3612236Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3612449Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3612663Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3612883Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3613101Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3613315Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3613539Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3613760Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3613976Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3614199Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3614415Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3614629Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:03:54.3614689Z ok (55.650s) 2022-11-23T03:03:54.3615130Z test_mixture_of_experts_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T03:03:54.3615588Z test_mixture_of_experts_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:54.3616040Z test_mixture_of_experts_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:54.3616465Z test_mixture_of_experts_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3616885Z test_mixture_of_experts_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3617363Z test_mixture_of_experts_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3617803Z test_mixture_of_experts_with_delay_before_free_offload_false_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3618238Z test_mixture_of_experts_with_delay_before_free_offload_false_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3618691Z test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3619133Z test_mixture_of_experts_with_delay_before_free_offload_true_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3619572Z test_mixture_of_experts_with_delay_before_free_offload_true_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3620017Z test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3620451Z test_nested_always_wrap_model_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3620877Z test_nested_always_wrap_model_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3621322Z test_nested_always_wrap_model_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3621756Z test_nested_always_wrap_model_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3622179Z test_nested_always_wrap_model_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3622654Z test_nested_always_wrap_model_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3623069Z test_nested_wrapped_model_offload_false_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3623471Z test_nested_wrapped_model_offload_false_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3623886Z test_nested_wrapped_model_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3624293Z test_nested_wrapped_model_offload_true_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3624736Z test_nested_wrapped_model_offload_true_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3625153Z test_nested_wrapped_model_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3625647Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3626104Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3626580Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3627042Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3627496Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3627965Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3628389Z test_transformer_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3628799Z test_transformer_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3629223Z test_transformer_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3629634Z test_transformer_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3630079Z test_transformer_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3630500Z test_transformer_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:03:54.3630508Z 2022-11-23T03:03:54.3630705Z ---------------------------------------------------------------------- 2022-11-23T03:03:54.3630778Z Ran 59 tests in 273.490s 2022-11-23T03:03:54.3630783Z 2022-11-23T03:03:54.3630849Z OK (skipped=54) 2022-11-23T03:03:54.3630853Z 2022-11-23T03:03:54.3630928Z Generating XML reports... 2022-11-23T03:03:54.3631257Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20221123025919.xml 2022-11-23T03:03:54.3631560Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20221123025919.xml 2022-11-23T03:03:54.3631859Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20221123025919.xml 2022-11-23T03:03:54.3632197Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20221123025919.xml 2022-11-23T03:03:54.3632211Z 2022-11-23T03:03:54.3632497Z ##[endgroup] 2022-11-23T03:03:54.3632862Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_core_ck6uj0_r) 2022-11-23T03:03:54.3632867Z 2022-11-23T03:03:54.3633061Z Running distributed/fsdp/test_fsdp_comm ... [2022-11-23 03:03:54.278728] 2022-11-23T03:03:54.3633470Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:03:54.279151] 2022-11-23T03:03:57.7811427Z 2022-11-23T03:03:57.7812320Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm 2022-11-23T03:03:57.7814616Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_comm_qttjwa84) 2022-11-23T03:03:57.7816744Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm 2022-11-23T03:03:57.7817482Z 2022-11-23T03:03:57.7817732Z Running tests... 2022-11-23T03:03:57.7818869Z ---------------------------------------------------------------------- 2022-11-23T03:03:57.7820143Z test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication) 2022-11-23T03:03:57.7822350Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.498s) 2022-11-23T03:03:57.7824102Z test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-11-23T03:03:57.7826374Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:57.7828015Z test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication) 2022-11-23T03:03:57.7830130Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:57.7831868Z test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-11-23T03:03:57.7834093Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:57.7836463Z test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication) 2022-11-23T03:03:57.7838623Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:57.7840395Z test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-11-23T03:03:57.7842598Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:57.7844213Z test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication) 2022-11-23T03:03:57.7846315Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:57.7848062Z test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-11-23T03:03:57.7850440Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:03:57.7851438Z 2022-11-23T03:03:57.7852187Z ---------------------------------------------------------------------- 2022-11-23T03:03:57.7853040Z Ran 8 tests in 0.511s 2022-11-23T03:03:57.7853440Z 2022-11-23T03:03:57.7853682Z OK (skipped=8) 2022-11-23T03:03:57.7854051Z 2022-11-23T03:03:57.7854372Z Generating XML reports... 2022-11-23T03:03:57.7856008Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221123030355.xml 2022-11-23T03:03:57.7856929Z 2022-11-23T03:03:57.7857682Z ##[endgroup] 2022-11-23T03:03:57.7859294Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_comm_qttjwa84) 2022-11-23T03:03:57.7860197Z 2022-11-23T03:03:57.7860963Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-11-23 03:03:57.781930] 2022-11-23T03:03:57.7863101Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:03:57.782891] 2022-11-23T03:04:01.1990648Z 2022-11-23T03:04:01.1991714Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_checkpoint 2022-11-23T03:04:01.1994295Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_72p6klm8) 2022-11-23T03:04:01.1996883Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint 2022-11-23T03:04:01.1997686Z 2022-11-23T03:04:01.1997946Z Running tests... 2022-11-23T03:04:01.1999100Z ---------------------------------------------------------------------- 2022-11-23T03:04:01.2001788Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.484s) 2022-11-23T03:04:01.2005114Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:04:01.2008441Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:04:01.2012235Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:04:01.2015557Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:04:01.2018833Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:04:01.2022120Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:04:01.2025590Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:04:01.2028897Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:01.2032219Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:01.2035559Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.005s) 2022-11-23T03:04:01.2038939Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:04:01.2042220Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:01.2045481Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:01.2048743Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:01.2052012Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:01.2053329Z 2022-11-23T03:04:01.2054087Z ---------------------------------------------------------------------- 2022-11-23T03:04:01.2055082Z Ran 16 tests in 0.512s 2022-11-23T03:04:01.2055502Z 2022-11-23T03:04:01.2055755Z OK (skipped=16) 2022-11-23T03:04:01.2056143Z 2022-11-23T03:04:01.2056442Z Generating XML reports... 2022-11-23T03:04:01.2058187Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123030359.xml 2022-11-23T03:04:01.2059167Z 2022-11-23T03:04:01.2059960Z ##[endgroup] 2022-11-23T03:04:01.2063684Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_72p6klm8) 2022-11-23T03:04:01.2064767Z 2022-11-23T03:04:01.2065626Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-11-23 03:04:01.200103] 2022-11-23T03:04:01.2067859Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:04:01.201081] 2022-11-23T03:04:04.7145382Z 2022-11-23T03:04:04.7147919Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint 2022-11-23T03:04:04.7152359Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_distributed_checkpoint_93cc6ops) 2022-11-23T03:04:04.7157198Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint 2022-11-23T03:04:04.7158719Z 2022-11-23T03:04:04.7159178Z Running tests... 2022-11-23T03:04:04.7161124Z ---------------------------------------------------------------------- 2022-11-23T03:04:04.7165427Z test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.502s) 2022-11-23T03:04:04.7170848Z test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:04.7172741Z 2022-11-23T03:04:04.7173828Z ---------------------------------------------------------------------- 2022-11-23T03:04:04.7175037Z Ran 2 tests in 0.503s 2022-11-23T03:04:04.7175626Z 2022-11-23T03:04:04.7175976Z OK (skipped=2) 2022-11-23T03:04:04.7176508Z 2022-11-23T03:04:04.7176934Z Generating XML reports... 2022-11-23T03:04:04.7179609Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123030402.xml 2022-11-23T03:04:04.7181106Z 2022-11-23T03:04:04.7182249Z ##[endgroup] 2022-11-23T03:04:04.7184888Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_distributed_checkpoint_93cc6ops) 2022-11-23T03:04:04.7186384Z 2022-11-23T03:04:04.7187532Z Running distributed/elastic/utils/util_test ... [2022-11-23 03:04:04.715474] 2022-11-23T03:04:04.7190609Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:04:04.716602] 2022-11-23T03:04:08.1444709Z 2022-11-23T03:04:08.1445951Z Expand the folded group to see the log file of distributed/elastic/utils/util_test 2022-11-23T03:04:08.1450016Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-util_test_o3kmbodc) 2022-11-23T03:04:08.1453946Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test 2022-11-23T03:04:08.1455300Z 2022-11-23T03:04:08.1455701Z Running tests... 2022-11-23T03:04:08.1457741Z ---------------------------------------------------------------------- 2022-11-23T03:04:08.1461222Z test_get_all_rank_0 (__main__.StoreUtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.489s) 2022-11-23T03:04:08.1465641Z test_get_all_rank_n (__main__.StoreUtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:08.1469636Z test_synchronize (__main__.StoreUtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:08.1473149Z test_get_logger (__main__.UtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:04:08.1476506Z test_get_logger_custom_name (__main__.UtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:04:08.1479482Z test_get_logger_different (__main__.UtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:04:08.1482087Z test_get_logger_none (__main__.UtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:04:08.1483131Z 2022-11-23T03:04:08.1484294Z ---------------------------------------------------------------------- 2022-11-23T03:04:08.1485330Z Ran 7 tests in 0.493s 2022-11-23T03:04:08.1485825Z 2022-11-23T03:04:08.1486118Z OK (skipped=7) 2022-11-23T03:04:08.1486575Z 2022-11-23T03:04:08.1486924Z Generating XML reports... 2022-11-23T03:04:08.1488905Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221123030406.xml 2022-11-23T03:04:08.1491414Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221123030406.xml 2022-11-23T03:04:08.1492457Z 2022-11-23T03:04:08.1493424Z ##[endgroup] 2022-11-23T03:04:08.1495454Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-util_test_o3kmbodc) 2022-11-23T03:04:08.1496602Z 2022-11-23T03:04:08.1497606Z Running distributed/elastic/utils/distributed_test ... [2022-11-23 03:04:08.145512] 2022-11-23T03:04:08.1500241Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:04:08.146605] 2022-11-23T03:04:11.6170142Z 2022-11-23T03:04:11.6171781Z Expand the folded group to see the log file of distributed/elastic/utils/distributed_test 2022-11-23T03:04:11.6176164Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-distributed_test_kmixbofe) 2022-11-23T03:04:11.6180423Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test 2022-11-23T03:04:11.6181808Z 2022-11-23T03:04:11.6182296Z Running tests... 2022-11-23T03:04:11.6184302Z ---------------------------------------------------------------------- 2022-11-23T03:04:11.6187893Z test_create_store_multi (__main__.DistributedUtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.492s) 2022-11-23T03:04:11.6192172Z test_create_store_no_port_multi (__main__.DistributedUtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:11.6194851Z test_create_store_single_server (__main__.DistributedUtilTest) ... ok (0.010s) 2022-11-23T03:04:11.6198570Z test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:11.6202060Z test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:04:11.6205957Z test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:11.6209468Z test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:11.6210876Z 2022-11-23T03:04:11.6211974Z ---------------------------------------------------------------------- 2022-11-23T03:04:11.6213229Z Ran 7 tests in 0.506s 2022-11-23T03:04:11.6213822Z 2022-11-23T03:04:11.6214153Z OK (skipped=6) 2022-11-23T03:04:11.6214715Z 2022-11-23T03:04:11.6215148Z Generating XML reports... 2022-11-23T03:04:11.6217801Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20221123030409.xml 2022-11-23T03:04:11.6219319Z 2022-11-23T03:04:11.6220476Z ##[endgroup] 2022-11-23T03:04:11.6223159Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-distributed_test_kmixbofe) 2022-11-23T03:04:11.6224689Z 2022-11-23T03:04:11.6226200Z Running distributed/elastic/timer/local_timer_example ... [2022-11-23 03:04:11.618101] 2022-11-23T03:04:11.6229454Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:04:11.619203] 2022-11-23T03:04:15.0454119Z 2022-11-23T03:04:15.0455979Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_example 2022-11-23T03:04:15.0460307Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-local_timer_example_3yc1gf3r) 2022-11-23T03:04:15.0464573Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example 2022-11-23T03:04:15.0465995Z 2022-11-23T03:04:15.0466430Z Running tests... 2022-11-23T03:04:15.0468441Z ---------------------------------------------------------------------- 2022-11-23T03:04:15.0472173Z test_example_start_method_spawn (__main__.LocalTimerExample) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.486s) 2022-11-23T03:04:15.0476572Z test_torch_mp_example (__main__.LocalTimerExample) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:15.0478277Z 2022-11-23T03:04:15.0479662Z ---------------------------------------------------------------------- 2022-11-23T03:04:15.0481083Z Ran 2 tests in 0.487s 2022-11-23T03:04:15.0481717Z 2022-11-23T03:04:15.0482083Z OK (skipped=2) 2022-11-23T03:04:15.0482624Z 2022-11-23T03:04:15.0483037Z Generating XML reports... 2022-11-23T03:04:15.0485600Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221123030413.xml 2022-11-23T03:04:15.0487013Z 2022-11-23T03:04:15.0488122Z ##[endgroup] 2022-11-23T03:04:15.0490772Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-local_timer_example_3yc1gf3r) 2022-11-23T03:04:15.0492276Z 2022-11-23T03:04:15.0493521Z Running distributed/elastic/multiprocessing/api_test ... [2022-11-23 03:04:15.046431] 2022-11-23T03:04:15.0496764Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:04:15.047532] 2022-11-23T03:04:18.6450930Z 2022-11-23T03:04:18.6452272Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test 2022-11-23T03:04:18.6457178Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-multiprocessing-api_test_jtakeva7) 2022-11-23T03:04:18.6461550Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test 2022-11-23T03:04:18.6463140Z 2022-11-23T03:04:18.6463607Z Running tests... 2022-11-23T03:04:18.6465505Z ---------------------------------------------------------------------- 2022-11-23T03:04:18.6469084Z test_get_failures (__main__.RunProcResultsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.523s) 2022-11-23T03:04:18.6473263Z test_is_failed (__main__.RunProcResultsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6477734Z test_args_env_len_mismatch (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6481690Z test_binary (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6485279Z test_binary_exit (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6488836Z test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6492308Z test_binary_raises (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6495867Z test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6499406Z test_function (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6502991Z test_function_large_ret_val (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6505117Z test_function_raise (__main__.StartProcessesListTest) 2022-11-23T03:04:18.6507960Z run 2x copies of echo2, raise an exception on the first ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6511460Z test_function_with_tensor (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6515022Z test_invalid_log_dir (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6518799Z test_multiprocess_context_close (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6522523Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6526097Z test_pcontext_wait (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6529569Z test_subprocess_context_close (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6533316Z test_to_map (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6536845Z test_validate_full_rank (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:04:18.6540404Z test_void_function (__main__.StartProcessesListTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6543866Z test_args_env_len_mismatch (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6547272Z test_binary_exit (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6550752Z test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6554500Z test_binary_raises (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6557810Z test_function_large_ret_val (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6558536Z test_function_raise (__main__.StartProcessesTest) 2022-11-23T03:04:18.6559629Z run 2x copies of echo2, raise an exception on the first ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6560867Z test_function_with_tensor (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6562120Z test_invalid_log_dir (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6563448Z test_multiprocess_context_close (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6564812Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6566119Z test_pcontext_wait (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6567359Z test_subprocess_context_close (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6568619Z test_to_map (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6569836Z test_validate_full_rank (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:04:18.6571109Z test_void_function (__main__.StartProcessesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6572394Z test_from_str_bad_input (__main__.StdTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6573510Z test_from_value (__main__.StdTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:18.6574773Z test_from_value_map (__main__.StdTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:04:18.6575289Z 2022-11-23T03:04:18.6575732Z ---------------------------------------------------------------------- 2022-11-23T03:04:18.6576220Z Ran 38 tests in 0.555s 2022-11-23T03:04:18.6576446Z 2022-11-23T03:04:18.6576592Z OK (skipped=38) 2022-11-23T03:04:18.6576807Z 2022-11-23T03:04:18.6577007Z Generating XML reports... 2022-11-23T03:04:18.6578025Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221123030416.xml 2022-11-23T03:04:18.6579333Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221123030416.xml 2022-11-23T03:04:18.6580641Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221123030416.xml 2022-11-23T03:04:18.6581817Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221123030416.xml 2022-11-23T03:04:18.6582308Z 2022-11-23T03:04:18.6582791Z ##[endgroup] 2022-11-23T03:04:18.6583963Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-multiprocessing-api_test_jtakeva7) 2022-11-23T03:04:18.6584592Z 2022-11-23T03:04:18.6585011Z Running distributed/elastic/events/lib_test ... [2022-11-23 03:04:18.646420] 2022-11-23T03:04:18.6585895Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-11-23 03:04:18.647473] 2022-11-23T03:04:21.8085221Z 2022-11-23T03:04:21.8086757Z Expand the folded group to see the log file of distributed/elastic/events/lib_test 2022-11-23T03:04:21.8091055Z ##[group]PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-events-lib_test_k1kumpqh) 2022-11-23T03:04:21.8093596Z ============================= test session starts ============================== 2022-11-23T03:04:21.8096322Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:04:21.8097996Z cachedir: .pytest_cache 2022-11-23T03:04:21.8100846Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:04:21.8103229Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:04:21.8105988Z plugins: xdoctest-1.0.2, shard-0.1.2, flakefinder-1.1.0, xdist-3.0.2, hypothesis-5.35.1, rerunfailures-10.3 2022-11-23T03:04:21.8107743Z collecting ... collected 8 items 2022-11-23T03:04:21.8113500Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2022-11-23T03:04:21.8118090Z 2022-11-23T03:04:21.8118975Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%] 2022-11-23T03:04:21.8120782Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%] 2022-11-23T03:04:21.8122639Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%] 2022-11-23T03:04:21.8124648Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%] 2022-11-23T03:04:21.8127512Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%] 2022-11-23T03:04:21.8129686Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%] 2022-11-23T03:04:21.8131660Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%] 2022-11-23T03:04:21.8133585Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%] 2022-11-23T03:04:21.8134609Z 2022-11-23T03:04:21.8135192Z ============================== 8 passed in 1.48s =============================== 2022-11-23T03:04:21.8135915Z 2022-11-23T03:04:21.8137112Z ##[endgroup] 2022-11-23T03:04:21.8139811Z FINISHED PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-events-lib_test_k1kumpqh) 2022-11-23T03:04:21.8141295Z 2022-11-23T03:04:21.8142492Z Running distributed/checkpoint/test_traverse ... [2022-11-23 03:04:21.809484] 2022-11-23T03:04:21.8145800Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:04:21.810592] 2022-11-23T03:04:25.3012564Z 2022-11-23T03:04:25.3014379Z Expand the folded group to see the log file of distributed/checkpoint/test_traverse 2022-11-23T03:04:25.3018527Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_traverse_j2h_6e5n) 2022-11-23T03:04:25.3022368Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_traverse 2022-11-23T03:04:25.3023646Z 2022-11-23T03:04:25.3024114Z Running tests... 2022-11-23T03:04:25.3026159Z ---------------------------------------------------------------------- 2022-11-23T03:04:25.3029642Z test_get_element (__main__.TestTraverse) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.499s) 2022-11-23T03:04:25.3033532Z test_set_element (__main__.TestTraverse) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:25.3038138Z test_traverse_doesnt_ignore_intermediate_collections (__main__.TestTraverse) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:25.3041691Z test_traverse_nested_dict (__main__.TestTraverse) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:25.3044966Z test_traverse_nested_list (__main__.TestTraverse) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:25.3048213Z test_traverse_shallow (__main__.TestTraverse) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:25.3051546Z test_traverse_with_ordered_dict (__main__.TestTraverse) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:04:25.3052984Z 2022-11-23T03:04:25.3054032Z ---------------------------------------------------------------------- 2022-11-23T03:04:25.3055294Z Ran 7 tests in 0.505s 2022-11-23T03:04:25.3055893Z 2022-11-23T03:04:25.3056261Z OK (skipped=7) 2022-11-23T03:04:25.3056816Z 2022-11-23T03:04:25.3057268Z Generating XML reports... 2022-11-23T03:04:25.3059674Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20221123030423.xml 2022-11-23T03:04:25.3061015Z 2022-11-23T03:04:25.3062175Z ##[endgroup] 2022-11-23T03:04:25.3064718Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_traverse_j2h_6e5n) 2022-11-23T03:04:25.3066597Z 2022-11-23T03:04:25.3067926Z Running distributed/checkpoint/test_file_system_checkpoint_cpu ... [2022-11-23 03:04:25.302232] 2022-11-23T03:04:25.3071310Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:04:25.303318] 2022-11-23T03:10:07.2222812Z 2022-11-23T03:10:07.2228581Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint_cpu 2022-11-23T03:10:07.2231110Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_ni3kibps) 2022-11-23T03:10:07.2233553Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu 2022-11-23T03:10:07.2234416Z 2022-11-23T03:10:07.2234666Z Running tests... 2022-11-23T03:10:07.2235801Z ---------------------------------------------------------------------- 2022-11-23T03:10:07.2237668Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14293 2022-11-23T03:10:07.2239624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14294 2022-11-23T03:10:07.2241408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2242580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2245233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2247429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2249236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2251961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2253861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2256472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2258945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2260776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2262912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2265825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2268143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2271136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2272728Z ok (3.721s) 2022-11-23T03:10:07.2275200Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 50 2022-11-23T03:10:07.2277670Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14417 2022-11-23T03:10:07.2279577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14418 2022-11-23T03:10:07.2282286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2284050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2286499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2288783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2290604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2293289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2295248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2297832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2299870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2301657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2303811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2305851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2307703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2309610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2310770Z ok (3.133s) 2022-11-23T03:10:07.2312322Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 49 2022-11-23T03:10:07.2313725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14541 2022-11-23T03:10:07.2314896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14542 2022-11-23T03:10:07.2316732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2317921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2319538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2320778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2321939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2323632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2324815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2326386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2327647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2328796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2330103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2331426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2333240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2335155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2336164Z ok (3.132s) 2022-11-23T03:10:07.2337686Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 48 2022-11-23T03:10:07.2339084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14665 2022-11-23T03:10:07.2340259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14666 2022-11-23T03:10:07.2341922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2343260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2344848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2346075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2347230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2348931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2350105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2351668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2352900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2354053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2355351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2356807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2358789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2360130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2360532Z ok (3.031s) 2022-11-23T03:10:07.2361128Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 47 2022-11-23T03:10:07.2361678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14789 2022-11-23T03:10:07.2362159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14790 2022-11-23T03:10:07.2362815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2363286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2363921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2364422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2364889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2365566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2366050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2366668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2367169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2367636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2368150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2368669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2369373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2370116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2370515Z ok (3.029s) 2022-11-23T03:10:07.2371100Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 46 2022-11-23T03:10:07.2371711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14913 2022-11-23T03:10:07.2372178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14914 2022-11-23T03:10:07.2372837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2373304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2373917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2374406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2374848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2375506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2375968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2376586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2377067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2377567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2378079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2378585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2379293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2380034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2380441Z ok (3.044s) 2022-11-23T03:10:07.2381038Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 45 2022-11-23T03:10:07.2381585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15037 2022-11-23T03:10:07.2382056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15038 2022-11-23T03:10:07.2382706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2383155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2383770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2384252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2384705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2385367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2385828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2386450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2386921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2387375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2387886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2388402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2389109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2389898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2390302Z ok (3.131s) 2022-11-23T03:10:07.2390883Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 44 2022-11-23T03:10:07.2391435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15161 2022-11-23T03:10:07.2391902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15162 2022-11-23T03:10:07.2392550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2393013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2393625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2394112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2394566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2395214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2395759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2396479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2396964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2397419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2397928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2398450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2399150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2399894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2400298Z ok (3.133s) 2022-11-23T03:10:07.2400903Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 43 2022-11-23T03:10:07.2401452Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15285 2022-11-23T03:10:07.2401917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15286 2022-11-23T03:10:07.2402565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2403027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2403629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2404122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2404575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2405242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2405702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2406315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2406799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2407241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2407753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2408333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2409051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2409794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2410198Z ok (3.032s) 2022-11-23T03:10:07.2410796Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 42 2022-11-23T03:10:07.2411332Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15409 2022-11-23T03:10:07.2411797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15410 2022-11-23T03:10:07.2412445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2412913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2413531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2414017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2414520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2415186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2415634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2416250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2416737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2417193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2417709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2418224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2418938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2419817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2420218Z ok (3.131s) 2022-11-23T03:10:07.2420817Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 41 2022-11-23T03:10:07.2421369Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15533 2022-11-23T03:10:07.2421837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15534 2022-11-23T03:10:07.2422492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2422955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2423565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2424052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2424507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2425170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2425632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2426250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2426781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2427238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2427740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2428262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2428978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2429728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2430132Z ok (3.132s) 2022-11-23T03:10:07.2430735Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 40 2022-11-23T03:10:07.2431286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15657 2022-11-23T03:10:07.2431742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15658 2022-11-23T03:10:07.2432394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2432911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2433531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2434019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2434476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2435138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2435588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2436288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2436776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2437232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2437745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2438265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2438976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2439719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2440107Z ok (3.030s) 2022-11-23T03:10:07.2440706Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 39 2022-11-23T03:10:07.2441257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15781 2022-11-23T03:10:07.2441721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15782 2022-11-23T03:10:07.2442373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2442835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2443461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2443934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2444388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2445050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2445565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2446184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2446669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2447120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2447615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2448134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2448838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2449582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2449990Z ok (3.030s) 2022-11-23T03:10:07.2450592Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 38 2022-11-23T03:10:07.2451199Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15905 2022-11-23T03:10:07.2451664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15906 2022-11-23T03:10:07.2452307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2452767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2453385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2453878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2454333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2454998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2455463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2456067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2456550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2457005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2457519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2458034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2458746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2459490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2459885Z ok (3.032s) 2022-11-23T03:10:07.2460472Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 37 2022-11-23T03:10:07.2461025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16029 2022-11-23T03:10:07.2461488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16030 2022-11-23T03:10:07.2462138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2462607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2463224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2463758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2464200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2464866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2465323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2465941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2466423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2466879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2467393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2468091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2468654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2469392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2469790Z ok (3.031s) 2022-11-23T03:10:07.2470387Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 36 2022-11-23T03:10:07.2470936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16153 2022-11-23T03:10:07.2471406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16154 2022-11-23T03:10:07.2472055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2472508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2473129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2473615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2474070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2474728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2475191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2475807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2476386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2476840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2477357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2477871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2478584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2479327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2479728Z ok (3.030s) 2022-11-23T03:10:07.2480312Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 35 2022-11-23T03:10:07.2480864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16277 2022-11-23T03:10:07.2481329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16278 2022-11-23T03:10:07.2481982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2482508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2483127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2483617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2484074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2484724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2485186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2485805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2486297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2486754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2487279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2487848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2488553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2489304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2489705Z ok (3.032s) 2022-11-23T03:10:07.2490303Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 34 2022-11-23T03:10:07.2490854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16401 2022-11-23T03:10:07.2491325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16402 2022-11-23T03:10:07.2491984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2492459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2493071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2493561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2494015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2494684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2495145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2495768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2496260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2496708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2497238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2497751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2498459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2499201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2499610Z ok (3.131s) 2022-11-23T03:10:07.2500205Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 33 2022-11-23T03:10:07.2500809Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16525 2022-11-23T03:10:07.2501318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16526 2022-11-23T03:10:07.2501992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2502446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2503074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2503564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2504022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2504687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2505155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2505777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2506247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2506755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2507279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2507999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2508572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2509277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2509681Z ok (3.133s) 2022-11-23T03:10:07.2510277Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 32 2022-11-23T03:10:07.2510823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16649 2022-11-23T03:10:07.2511302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16650 2022-11-23T03:10:07.2511946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2512420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2513042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2513536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2513992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2514659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2515126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2515732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2516274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2516733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2517434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2517951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2518666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2519475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2519864Z ok (3.132s) 2022-11-23T03:10:07.2520467Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 31 2022-11-23T03:10:07.2521017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16773 2022-11-23T03:10:07.2521482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16774 2022-11-23T03:10:07.2522131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2522596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2523213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2523704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2524148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2524811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2525325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2525949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2526462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2526926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2527447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2528149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2528715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2529430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2529851Z ok (3.032s) 2022-11-23T03:10:07.2531089Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 30 2022-11-23T03:10:07.2531641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16897 2022-11-23T03:10:07.2532107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16898 2022-11-23T03:10:07.2532758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2533226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2533828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2534320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2534775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2535439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2535898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2536513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2536995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2537447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2537942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2538515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2539231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2539976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2540378Z ok (2.931s) 2022-11-23T03:10:07.2540978Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 29 2022-11-23T03:10:07.2541521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17021 2022-11-23T03:10:07.2541971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17022 2022-11-23T03:10:07.2542619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2543095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2543714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2544212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2544709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2545379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2545826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2546471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2546971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2547441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2547970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2548680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2549244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2549948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2550355Z ok (3.130s) 2022-11-23T03:10:07.2550943Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 28 2022-11-23T03:10:07.2551490Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17145 2022-11-23T03:10:07.2551951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17146 2022-11-23T03:10:07.2552603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2553067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2553687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2554172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2554611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2555269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2555726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2556401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2556935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2557392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2557902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2558427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2559126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2559879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2560285Z ok (3.031s) 2022-11-23T03:10:07.2560882Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 27 2022-11-23T03:10:07.2561435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17269 2022-11-23T03:10:07.2561930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17270 2022-11-23T03:10:07.2562622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2563148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2563797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2564288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2564768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2565439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2565906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2566532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2567021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2567483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2568004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2568514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2569236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2569999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2570412Z ok (2.932s) 2022-11-23T03:10:07.2571027Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 26 2022-11-23T03:10:07.2571585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17393 2022-11-23T03:10:07.2572063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17394 2022-11-23T03:10:07.2572725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2573177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2573793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2574282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2574736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2575402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2575934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2576553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2577025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2577482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2577991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2578504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2579211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2579961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2580367Z ok (2.932s) 2022-11-23T03:10:07.2580952Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 25 2022-11-23T03:10:07.2581541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17517 2022-11-23T03:10:07.2582007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17518 2022-11-23T03:10:07.2582661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2583129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2583748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2584239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2584696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2585350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2585816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2586437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2586925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2587382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2587895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2588411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2589103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2589852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2590257Z ok (3.131s) 2022-11-23T03:10:07.2590859Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 24 2022-11-23T03:10:07.2591409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17641 2022-11-23T03:10:07.2591874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17642 2022-11-23T03:10:07.2592524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2592990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2593595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2594161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2594615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2595282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2595745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2596417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2596931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2597377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2597892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2598416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2599123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2599923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2600328Z ok (3.132s) 2022-11-23T03:10:07.2600929Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 23 2022-11-23T03:10:07.2601467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17765 2022-11-23T03:10:07.2601930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17766 2022-11-23T03:10:07.2602581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2603048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2603674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2604160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2604617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2605275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2605726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2606338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2606831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2607284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2607802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2608319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2609028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2609757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2610164Z ok (3.132s) 2022-11-23T03:10:07.2610762Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 22 2022-11-23T03:10:07.2611310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17889 2022-11-23T03:10:07.2611773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17890 2022-11-23T03:10:07.2612488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2612953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2613555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2614053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2614514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2615178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2615643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2616261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2616744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2617198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2617696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2618254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2618972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2619712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2620111Z ok (3.233s) 2022-11-23T03:10:07.2620710Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 21 2022-11-23T03:10:07.2621263Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18013 2022-11-23T03:10:07.2621717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18014 2022-11-23T03:10:07.2622366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2639154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2639949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2640486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2640945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2641626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2642076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2642726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2643215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2643668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2644176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2644699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2645406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2646141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2646537Z ok (3.132s) 2022-11-23T03:10:07.2647130Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 20 2022-11-23T03:10:07.2647815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18137 2022-11-23T03:10:07.2648275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18138 2022-11-23T03:10:07.2648924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2649380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2649978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2650448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2650896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2651546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2651995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2652597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2653065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2653559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2654053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2654564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2655262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2655988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2656383Z ok (3.029s) 2022-11-23T03:10:07.2656969Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 19 2022-11-23T03:10:07.2657506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18261 2022-11-23T03:10:07.2657957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18262 2022-11-23T03:10:07.2658595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2659046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2659649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2660130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2660574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2661231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2661673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2662279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2662754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2663192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2663690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2664390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2664934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2665674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2666055Z ok (3.130s) 2022-11-23T03:10:07.2666647Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 18 2022-11-23T03:10:07.2667184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18385 2022-11-23T03:10:07.2667640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18386 2022-11-23T03:10:07.2668276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2668727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2669331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2669801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2670243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2670891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2671387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2671995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2672464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2672908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2673400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2674102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2674652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2675331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2675725Z ok (3.133s) 2022-11-23T03:10:07.2676373Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 17 2022-11-23T03:10:07.2676910Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18509 2022-11-23T03:10:07.2677367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18510 2022-11-23T03:10:07.2677994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2678445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2679054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2679559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2680014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2680681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2681141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2681739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2682216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2682662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2683168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2684020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2684575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2685282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2685681Z ok (3.031s) 2022-11-23T03:10:07.2686266Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 16 2022-11-23T03:10:07.2686804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18633 2022-11-23T03:10:07.2687268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18634 2022-11-23T03:10:07.2687923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2688391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2689008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2689535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2689975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2690633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2691083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2691699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2692180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2692637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2693145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2693649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2694355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2695097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2695493Z ok (3.132s) 2022-11-23T03:10:07.2696095Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 15 2022-11-23T03:10:07.2696653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18757 2022-11-23T03:10:07.2697114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18758 2022-11-23T03:10:07.2697767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2698212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2698830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2699313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2699766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2700416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2700872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2701479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2701982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2702426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2702938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2703457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2704174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2704928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2705343Z ok (3.131s) 2022-11-23T03:10:07.2705930Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 14 2022-11-23T03:10:07.2706486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18881 2022-11-23T03:10:07.2706958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18882 2022-11-23T03:10:07.2707611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2708125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2708751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2709236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2709684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2710327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2710784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2711399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2711879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2712336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2712845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2713361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2714056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2714791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2715187Z ok (3.238s) 2022-11-23T03:10:07.2715783Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 13 2022-11-23T03:10:07.2716404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19005 2022-11-23T03:10:07.2716873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19006 2022-11-23T03:10:07.2717524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2717972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2718584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2719064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2719511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2720169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2720673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2721287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2721774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2722210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2722715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2723230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2723929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2724674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2725081Z ok (3.233s) 2022-11-23T03:10:07.2725674Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 12 2022-11-23T03:10:07.2726250Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19129 2022-11-23T03:10:07.2726714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19130 2022-11-23T03:10:07.2727354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2727816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2728432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2728915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2729369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2730008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2730461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2731075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2731551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2731996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2732505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2733024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2733728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2734458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2734855Z ok (3.433s) 2022-11-23T03:10:07.2735449Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 11 2022-11-23T03:10:07.2735988Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19253 2022-11-23T03:10:07.2736453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19254 2022-11-23T03:10:07.2737098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2737560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2738162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2738690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2739137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2739796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2740252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2740858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2741337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2741775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2742278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2742793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2743497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2744300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2744706Z ok (3.134s) 2022-11-23T03:10:07.2745304Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 10 2022-11-23T03:10:07.2745839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19377 2022-11-23T03:10:07.2746289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19378 2022-11-23T03:10:07.2746925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2747387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2748004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2748489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2748946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2749602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2750048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2750658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2751131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2751582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2752093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2752604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2753316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2754055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2754440Z ok (3.332s) 2022-11-23T03:10:07.2755023Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 9 2022-11-23T03:10:07.2755559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19501 2022-11-23T03:10:07.2756026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19502 2022-11-23T03:10:07.2756775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2757239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2757858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2758333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2758787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2759442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2759888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2760507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2760989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2761444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2761939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2762507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2763214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2763949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2764351Z ok (3.435s) 2022-11-23T03:10:07.2764946Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 8 2022-11-23T03:10:07.2765493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19625 2022-11-23T03:10:07.2765962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19626 2022-11-23T03:10:07.2766599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2767060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2767672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2768155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2768607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2769263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2769726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2770333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2770812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2771256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2771759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2772278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2772989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2773727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2774118Z ok (3.035s) 2022-11-23T03:10:07.2774713Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 7 2022-11-23T03:10:07.2775298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19749 2022-11-23T03:10:07.2775757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19750 2022-11-23T03:10:07.2776407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2776868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2777479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2777955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2778389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2779045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2779506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2780116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2780591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2781079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2781583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2782282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2782837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2783536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2783934Z ok (3.134s) 2022-11-23T03:10:07.2784525Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 6 2022-11-23T03:10:07.2785067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19873 2022-11-23T03:10:07.2785529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19874 2022-11-23T03:10:07.2786178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2786628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2787243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2787722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2788165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2788822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2789284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2789899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2790370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2790814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2791317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2791832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2792541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2793338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2793735Z ok (3.434s) 2022-11-23T03:10:07.2794318Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 5 2022-11-23T03:10:07.2794862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19997 2022-11-23T03:10:07.2795323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19998 2022-11-23T03:10:07.2795974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2796520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2797138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2797631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2798075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2798720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2799228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2799856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2800334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2800782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2801295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2801805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2802505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2803245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2803644Z ok (3.035s) 2022-11-23T03:10:07.2804233Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 4 2022-11-23T03:10:07.2804776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20121 2022-11-23T03:10:07.2805237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20122 2022-11-23T03:10:07.2805883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2806342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2806945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2807431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2807882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2808538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2808996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2809607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2810087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2810528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2811027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2811776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2812330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2813033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2813430Z ok (3.334s) 2022-11-23T03:10:07.2814021Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 3 2022-11-23T03:10:07.2814563Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20245 2022-11-23T03:10:07.2815026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20246 2022-11-23T03:10:07.2815662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2816130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2816747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2817273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2817721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2818373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2818815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2819426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2819902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2820356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2820867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2821390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2822095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2822826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2823208Z ok (3.431s) 2022-11-23T03:10:07.2823809Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 2 2022-11-23T03:10:07.2824352Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20369 2022-11-23T03:10:07.2824817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20370 2022-11-23T03:10:07.2825463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2825915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2826530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2827156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2827610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2828265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2828713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2829320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2829835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2830279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2830770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2831287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2831985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2832718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2833120Z ok (3.136s) 2022-11-23T03:10:07.2833712Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 1 2022-11-23T03:10:07.2834253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20493 2022-11-23T03:10:07.2834709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20494 2022-11-23T03:10:07.2835393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2835852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2836530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2837007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2837453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2838103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2838557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2839162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2839640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2840088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2840594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2841113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2841815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2842548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2842943Z ok (3.435s) 2022-11-23T03:10:07.2843560Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 0 2022-11-23T03:10:07.2844120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20617 2022-11-23T03:10:07.2844589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20618 2022-11-23T03:10:07.2845230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2845693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2846305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2846785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2847223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2847934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2848392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2849003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2849479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2849926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2850435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2850949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2851644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2852386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2852776Z ok (3.237s) 2022-11-23T03:10:07.2853409Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 50 2022-11-23T03:10:07.2854025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20741 2022-11-23T03:10:07.2854491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20742 2022-11-23T03:10:07.2855140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2855590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2856203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2856683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2857134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2857787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2858243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2858852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2859323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2859761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2860268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2860786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2861494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2862226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2862618Z ok (3.533s) 2022-11-23T03:10:07.2863249Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 49 2022-11-23T03:10:07.2863798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20865 2022-11-23T03:10:07.2864268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20866 2022-11-23T03:10:07.2864912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2865370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2865986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2866513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2866968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2867627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2868073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2868686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2869168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2869617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2870118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2870831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2871385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2872133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2872523Z ok (3.333s) 2022-11-23T03:10:07.2873159Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 48 2022-11-23T03:10:07.2873720Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20989 2022-11-23T03:10:07.2874194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20990 2022-11-23T03:10:07.2874844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2875310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2875922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2876447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2876901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2877556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2878019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2878634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2879116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2879567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2880062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2880581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2881283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2882030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2882413Z ok (3.636s) 2022-11-23T03:10:07.2883040Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 47 2022-11-23T03:10:07.2883601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21113 2022-11-23T03:10:07.2884063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21114 2022-11-23T03:10:07.2884772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2885227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2885841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2886320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2886766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2887410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2887866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2888470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2888936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2889387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2889950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2890473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2891179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2891915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2892310Z ok (3.637s) 2022-11-23T03:10:07.2892930Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 46 2022-11-23T03:10:07.2893503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21237 2022-11-23T03:10:07.2893963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21238 2022-11-23T03:10:07.2894609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2895076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2895688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2896155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2896604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2897249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2897703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2898317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2898797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2899244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2899748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2900268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2900972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2901694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2902084Z ok (3.334s) 2022-11-23T03:10:07.2902762Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 45 2022-11-23T03:10:07.2903327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21361 2022-11-23T03:10:07.2903792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21362 2022-11-23T03:10:07.2904438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2904898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2905502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2905978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2906424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2907083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2907534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2908188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2908672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2909121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2909613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2910122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2910828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2911569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2911959Z ok (3.237s) 2022-11-23T03:10:07.2912588Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 44 2022-11-23T03:10:07.2913156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21485 2022-11-23T03:10:07.2913624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21486 2022-11-23T03:10:07.2914261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2914722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2915341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2915819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2916324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2916976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2917434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2918032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2918512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2918954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2919459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2919979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2920736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2921475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2921863Z ok (3.633s) 2022-11-23T03:10:07.2922484Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 43 2022-11-23T03:10:07.2923054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21609 2022-11-23T03:10:07.2923515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21610 2022-11-23T03:10:07.2924152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2924613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2925231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2925711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2926153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2927018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2927479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2928095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2928573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2929023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2929529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2930225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2930770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2931476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2931875Z ok (3.734s) 2022-11-23T03:10:07.2932506Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 42 2022-11-23T03:10:07.2933069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21733 2022-11-23T03:10:07.2933533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21734 2022-11-23T03:10:07.2934177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2934629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2935246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2935757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2936215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2936972Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2937515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2938196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2938700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2939284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2939867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2940472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2941259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2942077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2942544Z ok (3.537s) 2022-11-23T03:10:07.2943252Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 41 2022-11-23T03:10:07.2943851Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21857 2022-11-23T03:10:07.2944397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21858 2022-11-23T03:10:07.2945135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2945671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2946406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2946964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2947495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2948231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2948710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2959802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2960331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2960786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2961301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2961822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2962582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2963327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2963711Z ok (3.638s) 2022-11-23T03:10:07.2964335Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 40 2022-11-23T03:10:07.2964903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21981 2022-11-23T03:10:07.2965369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21982 2022-11-23T03:10:07.2966021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2966478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2967087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2967591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2968035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2968688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2969285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2969901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2970378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2970828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2971329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2971842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2972526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2973260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2973648Z ok (3.637s) 2022-11-23T03:10:07.2974271Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 39 2022-11-23T03:10:07.2974836Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22105 2022-11-23T03:10:07.2975349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22106 2022-11-23T03:10:07.2975998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2976447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2977059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2977532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2977974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2978633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2979082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2979691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2980161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2980599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2981097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2981607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2982310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2983049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2983437Z ok (3.437s) 2022-11-23T03:10:07.2984063Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 38 2022-11-23T03:10:07.2984625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22229 2022-11-23T03:10:07.2985077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22230 2022-11-23T03:10:07.2985717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2986170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2986785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2987261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2987758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.2988412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2988856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2989464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2989935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2990381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2990883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.2991389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.2992097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2992833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.2993218Z ok (3.639s) 2022-11-23T03:10:07.2993890Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 37 2022-11-23T03:10:07.2994459Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22353 2022-11-23T03:10:07.2994917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22354 2022-11-23T03:10:07.2995557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2996018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2996700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2997175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.2997617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.2998280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.2998735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.2999346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.2999821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3000275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3000783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3001288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3001985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3002724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3003110Z ok (3.737s) 2022-11-23T03:10:07.3003743Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 36 2022-11-23T03:10:07.3004304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22477 2022-11-23T03:10:07.3004764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22478 2022-11-23T03:10:07.3005400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3005979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3006588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3007068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3007518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3008168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3008612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3009217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3009691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3010129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3010631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3011145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3011888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3012630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3013014Z ok (3.236s) 2022-11-23T03:10:07.3013636Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 35 2022-11-23T03:10:07.3014186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22601 2022-11-23T03:10:07.3014652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22602 2022-11-23T03:10:07.3015290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3015745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3016355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3016833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3017275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3017922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3018370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3018968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3019444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3019887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3020391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3020910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3021600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3022331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3022720Z ok (3.435s) 2022-11-23T03:10:07.3023350Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 34 2022-11-23T03:10:07.3023954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22725 2022-11-23T03:10:07.3024414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22726 2022-11-23T03:10:07.3025066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3025527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3026133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3026603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3027050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3027712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3028163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3028774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3029247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3029756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3030264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3030767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3031471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3032208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3032599Z ok (3.238s) 2022-11-23T03:10:07.3033224Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 33 2022-11-23T03:10:07.3033785Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22849 2022-11-23T03:10:07.3034249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22850 2022-11-23T03:10:07.3034877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3035333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3035943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3036488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3036931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3037588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3038046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3038655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3039127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3039573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3040077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3040588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3041284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3042075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3042472Z ok (3.738s) 2022-11-23T03:10:07.3043089Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 32 2022-11-23T03:10:07.3043647Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22973 2022-11-23T03:10:07.3044104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22974 2022-11-23T03:10:07.3044743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3045200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3045810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3046290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3046737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3047379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3047906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3048522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3048999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3049441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3049946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3050649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3051202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3051895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3052293Z ok (3.639s) 2022-11-23T03:10:07.3052923Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 31 2022-11-23T03:10:07.3053475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23097 2022-11-23T03:10:07.3053940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23098 2022-11-23T03:10:07.3054580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3055036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3055640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3056114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3056561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3057215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3057664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3058274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3058749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3059184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3059743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3060298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3061007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3061745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3062133Z ok (3.636s) 2022-11-23T03:10:07.3062756Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 30 2022-11-23T03:10:07.3063312Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23221 2022-11-23T03:10:07.3063761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23222 2022-11-23T03:10:07.3064394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3064852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3065464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3065981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3066426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3067080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3067529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3068129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3068608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3069053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3069551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3070058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3070758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3071541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3071922Z ok (3.639s) 2022-11-23T03:10:07.3072549Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 29 2022-11-23T03:10:07.3073113Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23345 2022-11-23T03:10:07.3073571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23346 2022-11-23T03:10:07.3074204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3074656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3075265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3075732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3076229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3076878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3077328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3077930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3078456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3078683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3078932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3079179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3079612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3080037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3080123Z ok (3.537s) 2022-11-23T03:10:07.3080602Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 28 2022-11-23T03:10:07.3080821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23469 2022-11-23T03:10:07.3081034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23470 2022-11-23T03:10:07.3081481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3081650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3082057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3082239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3082466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3082856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3083023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3083420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3083601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3083827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3084072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3084315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3084742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3085166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3085252Z ok (3.636s) 2022-11-23T03:10:07.3085745Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 27 2022-11-23T03:10:07.3085961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23593 2022-11-23T03:10:07.3086177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23594 2022-11-23T03:10:07.3086576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3086738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3087143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3087324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3087551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3087994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3088161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3088563Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3088743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3088967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3089215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3089459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3089889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3090318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3090404Z ok (3.137s) 2022-11-23T03:10:07.3090937Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 26 2022-11-23T03:10:07.3091154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23717 2022-11-23T03:10:07.3091367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23718 2022-11-23T03:10:07.3091763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3091931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3092336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3092523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3092747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3093131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3093301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3093703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3093883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3094106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3094348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3094588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3095021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3095449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3095538Z ok (3.437s) 2022-11-23T03:10:07.3096029Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 25 2022-11-23T03:10:07.3096243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23841 2022-11-23T03:10:07.3096454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23842 2022-11-23T03:10:07.3096853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3097022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3097472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3097658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3097885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3098271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3098437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3098836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3099018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3099232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3099474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3099905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3100143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3100610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3100702Z ok (3.335s) 2022-11-23T03:10:07.3101195Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 24 2022-11-23T03:10:07.3101409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23965 2022-11-23T03:10:07.3101621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23966 2022-11-23T03:10:07.3102012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3102181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3102585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3102771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3102990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3103381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3103546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3103947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3104130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3104355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3104604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3104853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3105282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3105699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3105786Z ok (3.736s) 2022-11-23T03:10:07.3106275Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 23 2022-11-23T03:10:07.3106490Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24089 2022-11-23T03:10:07.3106751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24090 2022-11-23T03:10:07.3107149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3107315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3107720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3107902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3108128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3108515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3108682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3109080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3109264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3109486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3109774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3110019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3110448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3110874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3110960Z ok (3.633s) 2022-11-23T03:10:07.3111447Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 22 2022-11-23T03:10:07.3111668Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24213 2022-11-23T03:10:07.3111872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24214 2022-11-23T03:10:07.3112273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3112441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3112843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3113026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3113248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3113637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3113804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3114201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3114382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3114605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3114849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3115089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3115517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3115943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3116122Z ok (3.436s) 2022-11-23T03:10:07.3116616Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 21 2022-11-23T03:10:07.3116832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24337 2022-11-23T03:10:07.3117048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24338 2022-11-23T03:10:07.3117445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3117613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3118021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3118197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3118420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3118813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3118977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3119426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3119611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3119833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3120076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3120504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3120744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3121165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3121252Z ok (3.232s) 2022-11-23T03:10:07.3121750Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 20 2022-11-23T03:10:07.3121963Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24461 2022-11-23T03:10:07.3122177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24462 2022-11-23T03:10:07.3122570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3122737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3123141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3123329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3123555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3123948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3124117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3124510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3124689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3124912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3125158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3125397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3125897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3126323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3126414Z ok (3.136s) 2022-11-23T03:10:07.3126904Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 19 2022-11-23T03:10:07.3127115Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24585 2022-11-23T03:10:07.3127327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24586 2022-11-23T03:10:07.3127722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3127888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3128295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3128476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3128745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3129134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3129299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3129699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3129880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3130103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3130351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3130587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3131019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3131441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3131526Z ok (3.238s) 2022-11-23T03:10:07.3132015Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 18 2022-11-23T03:10:07.3132233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24709 2022-11-23T03:10:07.3132443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24710 2022-11-23T03:10:07.3132840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3133011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3133414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3133600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3133826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3134215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3134381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3134783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3134967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3135234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3135478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3135724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3136154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3136580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3136667Z ok (3.336s) 2022-11-23T03:10:07.3137150Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 17 2022-11-23T03:10:07.3137366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24833 2022-11-23T03:10:07.3137582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24834 2022-11-23T03:10:07.3137973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3138144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3138590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3138775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3139002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3139390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3139556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3139951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3140134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3140356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3140599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3140841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3141268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3141690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3141774Z ok (3.238s) 2022-11-23T03:10:07.3142261Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 16 2022-11-23T03:10:07.3142479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24957 2022-11-23T03:10:07.3142694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24958 2022-11-23T03:10:07.3143095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3143255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3143660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3143843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3144066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3144455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3144665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3145066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3145245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3145468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3145710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3145951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3146374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3146793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3146882Z ok (3.636s) 2022-11-23T03:10:07.3147375Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 15 2022-11-23T03:10:07.3147591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25081 2022-11-23T03:10:07.3147845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25082 2022-11-23T03:10:07.3148242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3148408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3148814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3148997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3149222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3149606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3149776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3150179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3150356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3150575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3150820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3151063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3151491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3151916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3152001Z ok (3.535s) 2022-11-23T03:10:07.3152491Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 14 2022-11-23T03:10:07.3152705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25205 2022-11-23T03:10:07.3152916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25206 2022-11-23T03:10:07.3153306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3153473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3153876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3154105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3154330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3154725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3154891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3155290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3155473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3155689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3155930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3156228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3156714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3157194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3157283Z ok (3.536s) 2022-11-23T03:10:07.3157774Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 13 2022-11-23T03:10:07.3157988Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25329 2022-11-23T03:10:07.3158201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25330 2022-11-23T03:10:07.3158596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3158762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3159172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3159355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3159585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3159973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3160141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3160547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3160728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3160954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3161202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3161443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3161875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3162295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3162380Z ok (3.638s) 2022-11-23T03:10:07.3162874Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 12 2022-11-23T03:10:07.3163088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25453 2022-11-23T03:10:07.3163301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25454 2022-11-23T03:10:07.3163749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3163916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3164321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3164505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3164728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3165119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3165285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3165685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3165872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3166100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3166342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3166813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3167059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3167480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3167568Z ok (3.637s) 2022-11-23T03:10:07.3168056Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 11 2022-11-23T03:10:07.3168270Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25577 2022-11-23T03:10:07.3168480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25578 2022-11-23T03:10:07.3168869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3169039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3169441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3169625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3169850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3170233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3170396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3170795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3170981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3171199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3171444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3171686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3172119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3172537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3172624Z ok (3.637s) 2022-11-23T03:10:07.3173115Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 10 2022-11-23T03:10:07.3173377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25701 2022-11-23T03:10:07.3173587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25702 2022-11-23T03:10:07.3173984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3174152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3174557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3174732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3174958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3175349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3175515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3175913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3176135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3176357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3176600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3176841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3177266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3177688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3177782Z ok (3.235s) 2022-11-23T03:10:07.3178266Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 9 2022-11-23T03:10:07.3178485Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25825 2022-11-23T03:10:07.3178698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25826 2022-11-23T03:10:07.3179095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3179261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3179662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3179844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3180072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3180463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3180629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3181021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3181203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3181426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3181671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3181913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3182341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3182813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3182899Z ok (3.642s) 2022-11-23T03:10:07.3183393Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 8 2022-11-23T03:10:07.3183607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25949 2022-11-23T03:10:07.3183819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25950 2022-11-23T03:10:07.3184213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3184377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3184784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3184971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3185195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3185627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3185793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3186197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3186379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3186603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3186839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3187076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3187508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3187933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3188018Z ok (3.334s) 2022-11-23T03:10:07.3188509Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 7 2022-11-23T03:10:07.3188722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26073 2022-11-23T03:10:07.3188932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26074 2022-11-23T03:10:07.3189322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3189489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3189899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3190079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3190306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3190696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3190859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3191256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3191438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3191658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3191945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3192185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3192617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3193041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3193127Z ok (3.538s) 2022-11-23T03:10:07.3193600Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 6 2022-11-23T03:10:07.3193817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26197 2022-11-23T03:10:07.3194031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26198 2022-11-23T03:10:07.3194428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3194591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3194998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3195241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3195468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3195858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3196025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3196480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3196660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3196886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3197129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3197373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3197797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3198229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3198315Z ok (3.637s) 2022-11-23T03:10:07.3198803Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 5 2022-11-23T03:10:07.3199019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26321 2022-11-23T03:10:07.3199236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26322 2022-11-23T03:10:07.3199630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3199792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3200373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3200559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3200783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3201175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3201339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3201794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3201977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3202204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3202449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3202690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3203121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3203548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3203634Z ok (3.535s) 2022-11-23T03:10:07.3204117Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 4 2022-11-23T03:10:07.3204335Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26445 2022-11-23T03:10:07.3204549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26446 2022-11-23T03:10:07.3204988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3205156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3205562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3205745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3205961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3206347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3206515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3206917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3207104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3207325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3207568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3207807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3208234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3208655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3208744Z ok (3.636s) 2022-11-23T03:10:07.3209228Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 3 2022-11-23T03:10:07.3209443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26569 2022-11-23T03:10:07.3209655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26570 2022-11-23T03:10:07.3210048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3210214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3210619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3210805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3211033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3211469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3211633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3212034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3212210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3212433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3212675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3213106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3213347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3213771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3213855Z ok (3.636s) 2022-11-23T03:10:07.3214382Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 2 2022-11-23T03:10:07.3214602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26693 2022-11-23T03:10:07.3214813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26694 2022-11-23T03:10:07.3215206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3215372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3215775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3215962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3216191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3216583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3216751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3217150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3217331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3217553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3217794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3218036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3218466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3218886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3218973Z ok (3.638s) 2022-11-23T03:10:07.3219460Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 1 2022-11-23T03:10:07.3219672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26817 2022-11-23T03:10:07.3219885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26818 2022-11-23T03:10:07.3220276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3220487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3220891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3221072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3221299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:10:07.3221683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:10:07.3221849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:10:07.3222246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:10:07.3222427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:10:07.3222648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:10:07.3222898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:10:07.3223138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:10:07.3223610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3224036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:10:07.3224120Z ok (3.437s) 2022-11-23T03:10:07.3224572Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 0 2022-11-23T03:10:07.3224972Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T03:10:07.3225591Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.005s) 2022-11-23T03:10:07.3226202Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:07.3226877Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:07.3226889Z 2022-11-23T03:10:07.3227172Z ---------------------------------------------------------------------- 2022-11-23T03:10:07.3227269Z Ran 106 tests in 338.840s 2022-11-23T03:10:07.3227286Z 2022-11-23T03:10:07.3227371Z OK (skipped=4) 2022-11-23T03:10:07.3227379Z 2022-11-23T03:10:07.3227487Z Generating XML reports... 2022-11-23T03:10:07.3228071Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221123030426.xml 2022-11-23T03:10:07.3228675Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221123030426.xml 2022-11-23T03:10:07.3229362Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123030426.xml 2022-11-23T03:10:07.3229370Z 2022-11-23T03:10:07.3229755Z ##[endgroup] 2022-11-23T03:10:07.3230370Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_ni3kibps) 2022-11-23T03:10:07.3230378Z 2022-11-23T03:10:07.3230687Z Running distributed/checkpoint/test_dedup_tensors ... [2022-11-23 03:10:07.228111] 2022-11-23T03:10:07.3231291Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:07.228969] 2022-11-23T03:10:10.6801565Z 2022-11-23T03:10:10.6803129Z Expand the folded group to see the log file of distributed/checkpoint/test_dedup_tensors 2022-11-23T03:10:10.6805868Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_dedup_tensors_gcqe7clh) 2022-11-23T03:10:10.6808192Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors 2022-11-23T03:10:10.6809058Z 2022-11-23T03:10:10.6809428Z Running tests... 2022-11-23T03:10:10.6810716Z ---------------------------------------------------------------------- 2022-11-23T03:10:10.6812916Z test_dedup_shards (__main__.TestDedupTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.486s) 2022-11-23T03:10:10.6813914Z 2022-11-23T03:10:10.6835351Z ---------------------------------------------------------------------- 2022-11-23T03:10:10.6836453Z Ran 1 test in 0.486s 2022-11-23T03:10:10.6836868Z 2022-11-23T03:10:10.6837115Z OK (skipped=1) 2022-11-23T03:10:10.6837498Z 2022-11-23T03:10:10.6837790Z Generating XML reports... 2022-11-23T03:10:10.6840149Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20221123031008.xml 2022-11-23T03:10:10.6841134Z 2022-11-23T03:10:10.6841956Z ##[endgroup] 2022-11-23T03:10:10.6843736Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_dedup_tensors_gcqe7clh) 2022-11-23T03:10:10.6844764Z 2022-11-23T03:10:10.6845524Z Running distributed/algorithms/test_join ... [2022-11-23 03:10:10.680882] 2022-11-23T03:10:10.6847600Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:10.681730] 2022-11-23T03:10:14.1262713Z 2022-11-23T03:10:14.1264078Z Expand the folded group to see the log file of distributed/algorithms/test_join 2022-11-23T03:10:14.1267156Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-test_join_r0ixvi9e) 2022-11-23T03:10:14.1269360Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join 2022-11-23T03:10:14.1270107Z 2022-11-23T03:10:14.1270355Z Running tests... 2022-11-23T03:10:14.1271474Z ---------------------------------------------------------------------- 2022-11-23T03:10:14.1272384Z test_join_kwargs (__main__.TestJoin) 2022-11-23T03:10:14.1274693Z Tests passing keyword arguments to the context manager. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.485s) 2022-11-23T03:10:14.1276437Z test_multiple_joinable_disable (__main__.TestJoin) 2022-11-23T03:10:14.1278383Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:14.1279692Z test_multiple_joinables (__main__.TestJoin) 2022-11-23T03:10:14.1281618Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:14.1283009Z test_multiple_joinables_throw (__main__.TestJoin) 2022-11-23T03:10:14.1285058Z Tests ``throw_on_early_termination=True`` for multiple ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:14.1286327Z test_single_joinable (__main__.TestJoin) 2022-11-23T03:10:14.1288177Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:14.1290034Z test_single_joinable_disable (__main__.TestJoin) 2022-11-23T03:10:14.1291867Z Tests ``enable=False`` for a single :class:`Joinable`. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:14.1293186Z test_single_joinable_main_hooks (__main__.TestJoin) 2022-11-23T03:10:14.1295015Z Tests the main hooks of a single :class:`Joinable`. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:14.1296312Z test_single_joinable_post_hooks (__main__.TestJoin) 2022-11-23T03:10:14.1298117Z Tests the post-hooks of a single :class:`Joinable`. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:14.1299392Z test_single_joinable_throw (__main__.TestJoin) 2022-11-23T03:10:14.1301184Z Tests ``throw_on_early_termination=True`` for a single ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:14.1302132Z 2022-11-23T03:10:14.1302853Z ---------------------------------------------------------------------- 2022-11-23T03:10:14.1303684Z Ran 9 tests in 0.491s 2022-11-23T03:10:14.1304243Z 2022-11-23T03:10:14.1304491Z OK (skipped=9) 2022-11-23T03:10:14.1304864Z 2022-11-23T03:10:14.1305153Z Generating XML reports... 2022-11-23T03:10:14.1306717Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221123031012.xml 2022-11-23T03:10:14.1307598Z 2022-11-23T03:10:14.1308384Z ##[endgroup] 2022-11-23T03:10:14.1310007Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-test_join_r0ixvi9e) 2022-11-23T03:10:14.1310939Z 2022-11-23T03:10:14.1311692Z Running distributed/_tensor/test_view_ops ... [2022-11-23 03:10:14.127022] 2022-11-23T03:10:14.1313746Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:14.127970] 2022-11-23T03:10:17.5517936Z 2022-11-23T03:10:17.5519759Z Expand the folded group to see the log file of distributed/_tensor/test_view_ops 2022-11-23T03:10:17.5522896Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_view_ops_s665v3nl) 2022-11-23T03:10:17.5524942Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_view_ops 2022-11-23T03:10:17.5525664Z 2022-11-23T03:10:17.5525910Z Running tests... 2022-11-23T03:10:17.5527077Z ---------------------------------------------------------------------- 2022-11-23T03:10:17.5528997Z test_view_groups (__main__.TestViewOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.494s) 2022-11-23T03:10:17.5532402Z test_view_ops (__main__.TestViewOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.008s) 2022-11-23T03:10:17.5533300Z 2022-11-23T03:10:17.5534041Z ---------------------------------------------------------------------- 2022-11-23T03:10:17.5534903Z Ran 2 tests in 0.502s 2022-11-23T03:10:17.5535289Z 2022-11-23T03:10:17.5535621Z OK (skipped=2) 2022-11-23T03:10:17.5536291Z 2022-11-23T03:10:17.5536823Z Generating XML reports... 2022-11-23T03:10:17.5538952Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20221123031015.xml 2022-11-23T03:10:17.5539820Z 2022-11-23T03:10:17.5540579Z ##[endgroup] 2022-11-23T03:10:17.5542185Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_view_ops_s665v3nl) 2022-11-23T03:10:17.5543090Z 2022-11-23T03:10:17.5544350Z Running distributed/_tensor/test_tensor_ops ... [2022-11-23 03:10:17.552489] 2022-11-23T03:10:17.5546448Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:17.553345] 2022-11-23T03:10:21.0906314Z 2022-11-23T03:10:21.0907700Z Expand the folded group to see the log file of distributed/_tensor/test_tensor_ops 2022-11-23T03:10:21.0911392Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_tensor_ops_wq074vxt) 2022-11-23T03:10:21.0913602Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tensor_ops 2022-11-23T03:10:21.0914323Z 2022-11-23T03:10:21.0914573Z Running tests... 2022-11-23T03:10:21.0915699Z ---------------------------------------------------------------------- 2022-11-23T03:10:21.0917891Z test_aten_contiguous (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.488s) 2022-11-23T03:10:21.0920147Z test_clone (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0922842Z test_contiguous (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0925075Z test_detach (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0927292Z test_empty_like (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0929518Z test_fill_inplace (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0931809Z test_fill_inplace_partial_sum (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0934099Z test_full_like (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0936278Z test_index (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T03:10:21.0938463Z test_inplace_op (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0940671Z test_ones_like (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0942941Z test_ones_like_partial_sum (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0945218Z test_op_out_variant (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0947455Z test_zero_inplace (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0949685Z test_zeros_like (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0951931Z test_zeros_like_partial_sum (__main__.DistTensorOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:21.0953081Z 2022-11-23T03:10:21.0953822Z ---------------------------------------------------------------------- 2022-11-23T03:10:21.0954679Z Ran 16 tests in 0.503s 2022-11-23T03:10:21.0955061Z 2022-11-23T03:10:21.0955306Z OK (skipped=16) 2022-11-23T03:10:21.0955687Z 2022-11-23T03:10:21.0955978Z Generating XML reports... 2022-11-23T03:10:21.0957742Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20221123031019.xml 2022-11-23T03:10:21.0958666Z 2022-11-23T03:10:21.0959437Z ##[endgroup] 2022-11-23T03:10:21.0961091Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_tensor_ops_wq074vxt) 2022-11-23T03:10:21.0962011Z 2022-11-23T03:10:21.0962800Z Running distributed/_tensor/test_pointwise_ops ... [2022-11-23 03:10:21.091160] 2022-11-23T03:10:21.0964926Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_pointwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:21.092044] 2022-11-23T03:10:24.5360732Z 2022-11-23T03:10:24.5361948Z Expand the folded group to see the log file of distributed/_tensor/test_pointwise_ops 2022-11-23T03:10:24.5365861Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_pointwise_ops_9o3c1ao0) 2022-11-23T03:10:24.5368309Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_pointwise_ops 2022-11-23T03:10:24.5369697Z 2022-11-23T03:10:24.5370144Z Running tests... 2022-11-23T03:10:24.5371352Z ---------------------------------------------------------------------- 2022-11-23T03:10:24.5373369Z test_activations (__main__.DistElementwiseOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.481s) 2022-11-23T03:10:24.5375142Z test_dropout (__main__.DistElementwiseOpsTest) ... skip: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494 (0.001s) 2022-11-23T03:10:24.5377559Z test_dropout_backward (__main__.DistElementwiseOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:24.5379955Z test_dropout_errors (__main__.DistElementwiseOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:24.5382232Z test_mul_out (__main__.DistElementwiseOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:24.5383192Z 2022-11-23T03:10:24.5383921Z ---------------------------------------------------------------------- 2022-11-23T03:10:24.5384760Z Ran 5 tests in 0.485s 2022-11-23T03:10:24.5385161Z 2022-11-23T03:10:24.5385433Z OK (skipped=5) 2022-11-23T03:10:24.5385801Z 2022-11-23T03:10:24.5386086Z Generating XML reports... 2022-11-23T03:10:24.5387819Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_pointwise_ops/TEST-DistElementwiseOpsTest-20221123031022.xml 2022-11-23T03:10:24.5388787Z 2022-11-23T03:10:24.5389551Z ##[endgroup] 2022-11-23T03:10:24.5391236Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_pointwise_ops_9o3c1ao0) 2022-11-23T03:10:24.5392177Z 2022-11-23T03:10:24.5392904Z Running distributed/_tensor/test_math_ops ... [2022-11-23 03:10:24.536509] 2022-11-23T03:10:24.5394960Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:24.537299] 2022-11-23T03:10:28.0243427Z 2022-11-23T03:10:28.0244577Z Expand the folded group to see the log file of distributed/_tensor/test_math_ops 2022-11-23T03:10:28.0248022Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_math_ops_x7uir87q) 2022-11-23T03:10:28.0250659Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_math_ops 2022-11-23T03:10:28.0251382Z 2022-11-23T03:10:28.0251631Z Running tests... 2022-11-23T03:10:28.0252921Z ---------------------------------------------------------------------- 2022-11-23T03:10:28.0254868Z test_softmax_fwd (__main__.DistMathOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.476s) 2022-11-23T03:10:28.0257110Z test_softmax_with_bwd (__main__.DistMathOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:10:28.0259280Z test_sum (__main__.DistMathOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:28.0260166Z 2022-11-23T03:10:28.0260902Z ---------------------------------------------------------------------- 2022-11-23T03:10:28.0261742Z Ran 3 tests in 0.480s 2022-11-23T03:10:28.0262146Z 2022-11-23T03:10:28.0262384Z OK (skipped=3) 2022-11-23T03:10:28.0262757Z 2022-11-23T03:10:28.0263040Z Generating XML reports... 2022-11-23T03:10:28.0264822Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20221123031026.xml 2022-11-23T03:10:28.0265753Z 2022-11-23T03:10:28.0266526Z ##[endgroup] 2022-11-23T03:10:28.0268125Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_math_ops_x7uir87q) 2022-11-23T03:10:28.0269017Z 2022-11-23T03:10:28.0269760Z Running distributed/_tensor/test_device_mesh ... [2022-11-23 03:10:28.024891] 2022-11-23T03:10:28.0271855Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_device_mesh.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:28.025733] 2022-11-23T03:10:31.5397530Z 2022-11-23T03:10:31.5398864Z Expand the folded group to see the log file of distributed/_tensor/test_device_mesh 2022-11-23T03:10:31.5401180Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_device_mesh_6cjso4id) 2022-11-23T03:10:31.5403332Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_device_mesh 2022-11-23T03:10:31.5404083Z 2022-11-23T03:10:31.5404395Z Running tests... 2022-11-23T03:10:31.5405521Z ---------------------------------------------------------------------- 2022-11-23T03:10:31.5407554Z test_all_gather_1d (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.513s) 2022-11-23T03:10:31.5409939Z test_all_gather_nd (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5412321Z test_all_gather_uneven (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5414705Z test_all_reduce_1d (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5417037Z test_all_reduce_nd (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5419328Z test_all_to_all_1d (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5421613Z test_all_to_all_nd (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5424473Z test_broadcast_1d (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5426823Z test_broadcast_nd (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5429200Z test_reduce_scatter_1d (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5431575Z test_reduce_scatter_nd (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5433997Z test_reduce_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5436556Z test_scatter_1d (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5439087Z test_scatter_nd (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5441428Z test_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5443719Z test_device_mesh_2d (__main__.DeviceMeshTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5446022Z test_device_mesh_2d_from_dim_groups (__main__.DeviceMeshTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5448385Z test_device_mesh_dim_groups_error (__main__.DeviceMeshTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5450649Z test_device_mesh_nd (__main__.DeviceMeshTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:31.5451559Z 2022-11-23T03:10:31.5452296Z ---------------------------------------------------------------------- 2022-11-23T03:10:31.5453149Z Ran 19 tests in 0.531s 2022-11-23T03:10:31.5453545Z 2022-11-23T03:10:31.5453787Z OK (skipped=19) 2022-11-23T03:10:31.5454161Z 2022-11-23T03:10:31.5454447Z Generating XML reports... 2022-11-23T03:10:31.5456154Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshCollectiveTest-20221123031029.xml 2022-11-23T03:10:31.5458274Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshTest-20221123031029.xml 2022-11-23T03:10:31.5459167Z 2022-11-23T03:10:31.5459930Z ##[endgroup] 2022-11-23T03:10:31.5461570Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_device_mesh_6cjso4id) 2022-11-23T03:10:31.5462482Z 2022-11-23T03:10:31.5463207Z Running distributed/_tensor/test_api ... [2022-11-23 03:10:31.540458] 2022-11-23T03:10:31.5465234Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_api.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:31.541302] 2022-11-23T03:10:35.0597923Z 2022-11-23T03:10:35.0599022Z Expand the folded group to see the log file of distributed/_tensor/test_api 2022-11-23T03:10:35.0601076Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_api_o5lod32y) 2022-11-23T03:10:35.0603473Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_api 2022-11-23T03:10:35.0604118Z 2022-11-23T03:10:35.0604347Z Running tests... 2022-11-23T03:10:35.0605370Z ---------------------------------------------------------------------- 2022-11-23T03:10:35.0607200Z test_distribute_module (__main__.DTensorAPITest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.482s) 2022-11-23T03:10:35.0609382Z test_distribute_module_input_fn_output_fn (__main__.DTensorAPITest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:10:35.0611487Z test_distribute_tensor (__main__.DTensorAPITest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:35.0613598Z test_distribute_tensor_errors (__main__.DTensorAPITest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:35.0615749Z test_distribute_tensor_uneven_sharding (__main__.DTensorAPITest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:35.0616663Z 2022-11-23T03:10:35.0617507Z ---------------------------------------------------------------------- 2022-11-23T03:10:35.0618275Z Ran 5 tests in 0.488s 2022-11-23T03:10:35.0618644Z 2022-11-23T03:10:35.0618895Z OK (skipped=5) 2022-11-23T03:10:35.0619236Z 2022-11-23T03:10:35.0619500Z Generating XML reports... 2022-11-23T03:10:35.0620937Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20221123031033.xml 2022-11-23T03:10:35.0621742Z 2022-11-23T03:10:35.0622439Z ##[endgroup] 2022-11-23T03:10:35.0623858Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_api_o5lod32y) 2022-11-23T03:10:35.0624654Z 2022-11-23T03:10:35.0625375Z Running distributed/_tensor/parallel/test_tp_style ... [2022-11-23 03:10:35.060391] 2022-11-23T03:10:35.0627339Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_tp_style.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:35.061246] 2022-11-23T03:10:38.5197304Z 2022-11-23T03:10:38.5198908Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_tp_style 2022-11-23T03:10:38.5201506Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_style (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_tp_style_ympveciz) 2022-11-23T03:10:38.5203745Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_tp_style 2022-11-23T03:10:38.5204604Z 2022-11-23T03:10:38.5204834Z Running tests... 2022-11-23T03:10:38.5205979Z ---------------------------------------------------------------------- 2022-11-23T03:10:38.5208121Z test_colwise_parallel_style (__main__.TensorParallelStyleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.497s) 2022-11-23T03:10:38.5210639Z test_make_input_replicate_1d (__main__.TensorParallelStyleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:38.5213094Z test_make_input_shard_1d (__main__.TensorParallelStyleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:10:38.5215569Z test_make_output_replicate_1d (__main__.TensorParallelStyleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:38.5218053Z test_make_output_shard_1d (__main__.TensorParallelStyleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:38.5220940Z test_make_output_tensor (__main__.TensorParallelStyleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:38.5223396Z test_prepare_output_error (__main__.TensorParallelStyleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:10:38.5225856Z test_rowwise_parallel_style (__main__.TensorParallelStyleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:38.5226854Z 2022-11-23T03:10:38.5227618Z ---------------------------------------------------------------------- 2022-11-23T03:10:38.5228451Z Ran 8 tests in 0.502s 2022-11-23T03:10:38.5228856Z 2022-11-23T03:10:38.5229096Z OK (skipped=8) 2022-11-23T03:10:38.5229472Z 2022-11-23T03:10:38.5229773Z Generating XML reports... 2022-11-23T03:10:38.5231587Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_tp_style/TEST-TensorParallelStyleTest-20221123031036.xml 2022-11-23T03:10:38.5232615Z 2022-11-23T03:10:38.5233426Z ##[endgroup] 2022-11-23T03:10:38.5235386Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_style (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_tp_style_ympveciz) 2022-11-23T03:10:38.5236608Z 2022-11-23T03:10:38.5237523Z Running distributed/_tensor/parallel/test_parallelize_api ... [2022-11-23 03:10:38.520308] 2022-11-23T03:10:38.5239796Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_parallelize_api.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:38.521151] 2022-11-23T03:10:41.9796023Z 2022-11-23T03:10:41.9797338Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_parallelize_api 2022-11-23T03:10:41.9800099Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_rx89_6e8) 2022-11-23T03:10:41.9802591Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api 2022-11-23T03:10:41.9803405Z 2022-11-23T03:10:41.9803680Z Running tests... 2022-11-23T03:10:41.9804859Z ---------------------------------------------------------------------- 2022-11-23T03:10:41.9806954Z test_creat_1d_device_mesh (__main__.TensorParallelAPITests) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.482s) 2022-11-23T03:10:41.9809470Z test_creat_1d_device_mesh_error (__main__.TensorParallelAPITests) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:41.9811917Z test_parallelize_mlp (__main__.TensorParallelAPITests) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:41.9814382Z test_parallelize_mlp_error (__main__.TensorParallelAPITests) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:41.9815393Z 2022-11-23T03:10:41.9816162Z ---------------------------------------------------------------------- 2022-11-23T03:10:41.9817022Z Ran 4 tests in 0.485s 2022-11-23T03:10:41.9817424Z 2022-11-23T03:10:41.9817666Z OK (skipped=4) 2022-11-23T03:10:41.9818037Z 2022-11-23T03:10:41.9818328Z Generating XML reports... 2022-11-23T03:10:41.9820169Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20221123031040.xml 2022-11-23T03:10:41.9821214Z 2022-11-23T03:10:41.9821992Z ##[endgroup] 2022-11-23T03:10:41.9823867Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_rx89_6e8) 2022-11-23T03:10:41.9825417Z 2022-11-23T03:10:41.9826210Z Running distributed/_shard/test_sharder ... [2022-11-23 03:10:41.980300] 2022-11-23T03:10:41.9828306Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:41.981160] 2022-11-23T03:10:44.9238838Z 2022-11-23T03:10:44.9240135Z Expand the folded group to see the log file of distributed/_shard/test_sharder 2022-11-23T03:10:44.9243334Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-test_sharder_af2j5f4n) 2022-11-23T03:10:44.9244906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxroga5pj 2022-11-23T03:10:44.9246378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxroga5pj/_remote_module_non_scriptable.py 2022-11-23T03:10:44.9247198Z 2022-11-23T03:10:44.9248086Z ##[endgroup] 2022-11-23T03:10:44.9250301Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-test_sharder_af2j5f4n) 2022-11-23T03:10:44.9251460Z 2022-11-23T03:10:44.9253682Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-11-23 03:10:44.924513] 2022-11-23T03:10:44.9261439Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:44.925374] 2022-11-23T03:10:48.3035331Z 2022-11-23T03:10:48.3037419Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-11-23T03:10:48.3041206Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_v2rz6myd) 2022-11-23T03:10:48.3043639Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops 2022-11-23T03:10:48.3044439Z 2022-11-23T03:10:48.3044686Z Running tests... 2022-11-23T03:10:48.3046272Z ---------------------------------------------------------------------- 2022-11-23T03:10:48.3048230Z test_clone (__main__.TestTensorOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.482s) 2022-11-23T03:10:48.3050401Z test_deep_copy (__main__.TestTensorOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:48.3052557Z test_detach (__main__.TestTensorOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:48.3055556Z test_inplace_copy (__main__.TestTensorOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:48.3057796Z test_set_requires_grad (__main__.TestTensorOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:48.3058722Z 2022-11-23T03:10:48.3059461Z ---------------------------------------------------------------------- 2022-11-23T03:10:48.3060302Z Ran 5 tests in 0.486s 2022-11-23T03:10:48.3060697Z 2022-11-23T03:10:48.3060935Z OK (skipped=5) 2022-11-23T03:10:48.3061306Z 2022-11-23T03:10:48.3061590Z Generating XML reports... 2022-11-23T03:10:48.3063304Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221123031046.xml 2022-11-23T03:10:48.3064272Z 2022-11-23T03:10:48.3065059Z ##[endgroup] 2022-11-23T03:10:48.3066914Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_v2rz6myd) 2022-11-23T03:10:48.3068357Z 2022-11-23T03:10:48.3069182Z Running distributed/_composable/test_fully_shard ... [2022-11-23 03:10:48.304246] 2022-11-23T03:10:48.3071357Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_fully_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:48.305153] 2022-11-23T03:10:51.7133259Z 2022-11-23T03:10:51.7134682Z Expand the folded group to see the log file of distributed/_composable/test_fully_shard 2022-11-23T03:10:51.7137772Z ##[group]PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_fully_shard_x8v4mb92) 2022-11-23T03:10:51.7139973Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_fully_shard 2022-11-23T03:10:51.7140727Z 2022-11-23T03:10:51.7140971Z Running tests... 2022-11-23T03:10:51.7142086Z ---------------------------------------------------------------------- 2022-11-23T03:10:51.7143171Z test_auto_wrap_policy (__main__.TestFSDPInitialization) 2022-11-23T03:10:51.7145233Z Tests passing an ``auto_wrap_policy``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.497s) 2022-11-23T03:10:51.7147010Z test_device_id (__main__.TestFSDPInitialization) 2022-11-23T03:10:51.7148803Z Tests passing a ``device_id``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:51.7150123Z test_materialize_meta_module (__main__.TestFSDPInitialization) 2022-11-23T03:10:51.7152019Z Tests materializing a meta-device module. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:51.7153572Z test_sync_module_states (__main__.TestFSDPInitialization) 2022-11-23T03:10:51.7155432Z Tests passing ``sync_module_states=True``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:51.7156960Z test_training (__main__.TestFSDPRuntime) 2022-11-23T03:10:51.7158753Z Tests training (forward, backward, optimizer). ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:51.7159689Z 2022-11-23T03:10:51.7160424Z ---------------------------------------------------------------------- 2022-11-23T03:10:51.7161265Z Ran 5 tests in 0.501s 2022-11-23T03:10:51.7161660Z 2022-11-23T03:10:51.7161895Z OK (skipped=5) 2022-11-23T03:10:51.7162248Z 2022-11-23T03:10:51.7162534Z Generating XML reports... 2022-11-23T03:10:51.7164282Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20221123031049.xml 2022-11-23T03:10:51.7166489Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20221123031049.xml 2022-11-23T03:10:51.7167393Z 2022-11-23T03:10:51.7168160Z ##[endgroup] 2022-11-23T03:10:51.7169858Z FINISHED PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_fully_shard_x8v4mb92) 2022-11-23T03:10:51.7170817Z 2022-11-23T03:10:51.7171611Z Running distributed/_composable/test_checkpoint ... [2022-11-23 03:10:51.713937] 2022-11-23T03:10:51.7173752Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:51.714853] 2022-11-23T03:10:55.1212186Z 2022-11-23T03:10:55.1213189Z Expand the folded group to see the log file of distributed/_composable/test_checkpoint 2022-11-23T03:10:55.1217038Z ##[group]PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_checkpoint_njm667xi) 2022-11-23T03:10:55.1219743Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_checkpoint 2022-11-23T03:10:55.1220495Z 2022-11-23T03:10:55.1220742Z Running tests... 2022-11-23T03:10:55.1221859Z ---------------------------------------------------------------------- 2022-11-23T03:10:55.1223811Z test_tensor_only_cpu (__main__.TestCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.015s) 2022-11-23T03:10:55.1226055Z test_tensor_only_gpu (__main__.TestCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:10:55.1226990Z 2022-11-23T03:10:55.1227767Z ---------------------------------------------------------------------- 2022-11-23T03:10:55.1228609Z Ran 2 tests in 0.016s 2022-11-23T03:10:55.1229012Z 2022-11-23T03:10:55.1229252Z OK (skipped=2) 2022-11-23T03:10:55.1229620Z 2022-11-23T03:10:55.1229913Z Generating XML reports... 2022-11-23T03:10:55.1231568Z Generated XML report: test-reports/python-unittest/distributed._composable.test_checkpoint/TEST-TestCheckpoint-20221123031053.xml 2022-11-23T03:10:55.1232492Z 2022-11-23T03:10:55.1233254Z ##[endgroup] 2022-11-23T03:10:55.1235147Z FINISHED PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_checkpoint_njm667xi) 2022-11-23T03:10:55.1236320Z 2022-11-23T03:10:56.5844643Z 2022-11-23T03:10:56.5845570Z real 93m33.653s 2022-11-23T03:10:56.5846727Z user 172m41.949s 2022-11-23T03:10:56.5847871Z sys 77m9.560s 2022-11-23T03:10:56.5848625Z + assert_git_not_dirty 2022-11-23T03:10:56.5849906Z + [[ linux-focal-rocm5.2-py3.8 != *rocm* ]] 2022-11-23T03:10:56.5850983Z + [[ linux-focal-rocm5.2-py3.8 == *cuda* ]] 2022-11-23T03:10:56.5851643Z + [[ 1 == 1 ]] 2022-11-23T03:10:56.5852205Z + test_rpc 2022-11-23T03:10:56.5853087Z + [[ linux-focal-rocm5.2-py3.8 != *rocm* ]] 2022-11-23T03:10:56.6005941Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2022-11-23T03:10:56.6006583Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2022-11-23T03:10:56.6007279Z docker exec -t "de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2022-11-23T03:10:56.6040133Z shell: /bin/bash -e {0} 2022-11-23T03:10:56.6040420Z env: 2022-11-23T03:10:56.6040678Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:10:56.6041035Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:10:56.6041603Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:10:56.6042226Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:10:56.6042625Z ##[endgroup] 2022-11-23T03:10:56.7504494Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2022-11-23T03:10:56.8062929Z Prepare all required actions 2022-11-23T03:10:56.8063993Z Getting action download info 2022-11-23T03:10:57.0315964Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2022-11-23T03:10:57.8900185Z ##[group]Run ./.github/actions/get-workflow-job-id 2022-11-23T03:10:57.8900502Z with: 2022-11-23T03:10:57.8901052Z github-token: *** 2022-11-23T03:10:57.8901302Z env: 2022-11-23T03:10:57.8901558Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:10:57.8901890Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:10:57.8902387Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:10:57.8902968Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:10:57.8903354Z ##[endgroup] 2022-11-23T03:10:57.8938336Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2022-11-23T03:10:57.8938833Z with: 2022-11-23T03:10:57.8939065Z shell: bash 2022-11-23T03:10:57.8939326Z timeout_minutes: 10 2022-11-23T03:10:57.8939594Z max_attempts: 5 2022-11-23T03:10:57.8939857Z retry_wait_seconds: 30 2022-11-23T03:10:57.8940415Z command: set -eux python3 -m pip install requests==2.26.0 GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}" 2022-11-23T03:10:57.8940974Z polling_interval_seconds: 1 2022-11-23T03:10:57.8941243Z warning_on_retry: true 2022-11-23T03:10:57.8941524Z continue_on_error: false 2022-11-23T03:10:57.8941787Z env: 2022-11-23T03:10:57.8942038Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:10:57.8942369Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:10:57.8942862Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:10:57.8943450Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:10:57.8943984Z GITHUB_TOKEN: *** 2022-11-23T03:10:57.8944239Z ##[endgroup] 2022-11-23T03:10:57.9699499Z + python3 -m pip install requests==2.26.0 2022-11-23T03:10:58.7769087Z Collecting requests==2.26.0 2022-11-23T03:10:58.8981465Z Using cached https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl 2022-11-23T03:10:58.9166387Z Collecting idna<4,>=2.5; python_version >= "3" (from requests==2.26.0) 2022-11-23T03:10:58.9425876Z Using cached https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl 2022-11-23T03:10:58.9471148Z Collecting certifi>=2017.4.17 (from requests==2.26.0) 2022-11-23T03:10:58.9842696Z Using cached https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl 2022-11-23T03:10:58.9892840Z Collecting urllib3<1.27,>=1.21.1 (from requests==2.26.0) 2022-11-23T03:10:59.0710255Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T03:10:59.0963052Z Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests==2.26.0) 2022-11-23T03:10:59.1795792Z Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2022-11-23T03:10:59.1833865Z Installing collected packages: idna, certifi, urllib3, charset-normalizer, requests 2022-11-23T03:10:59.3671119Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.27.1 urllib3-1.26.12 2022-11-23T03:10:59.4155102Z ++ python3 .github/scripts/get_workflow_job_id.py 3528394938 worker-rocm-amd-114 2022-11-23T03:11:02.1360948Z + GHA_WORKFLOW_JOB_ID=9655437735 2022-11-23T03:11:02.1362262Z + echo job-id=9655437735 2022-11-23T03:11:02.9729271Z Command completed after 1 attempt(s). 2022-11-23T03:11:02.9977650Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2022-11-23T03:11:02.9977994Z kill "$MONITOR_SCRIPT_PID" 2022-11-23T03:11:03.0010743Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T03:11:03.0011041Z env: 2022-11-23T03:11:03.0011299Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:11:03.0011629Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:11:03.0012145Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:11:03.0012715Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:11:03.0013102Z MONITOR_SCRIPT_PID: 18765 2022-11-23T03:11:03.0013355Z ##[endgroup] 2022-11-23T03:11:03.0243237Z Prepare all required actions 2022-11-23T03:11:03.0243741Z Getting action download info 2022-11-23T03:11:03.5149039Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2022-11-23T03:11:04.5804400Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb) 2022-11-23T03:11:05.4776350Z ##[group]Run ./.github/actions/upload-test-artifacts 2022-11-23T03:11:05.4776651Z with: 2022-11-23T03:11:05.4776887Z use-gha: true 2022-11-23T03:11:05.4777255Z file-suffix: test-distributed-1-2-linux.rocm.gpu_9655437735 2022-11-23T03:11:05.4777598Z env: 2022-11-23T03:11:05.4777852Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:11:05.4778159Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:11:05.4778671Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:11:05.4779290Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:11:05.4779665Z ##[endgroup] 2022-11-23T03:11:05.4852209Z ##[group]Run actions/upload-artifact@v3 2022-11-23T03:11:05.4852507Z with: 2022-11-23T03:11:05.4852937Z name: test-jsons-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437735.zip 2022-11-23T03:11:05.4853383Z retention-days: 14 2022-11-23T03:11:05.4853649Z if-no-files-found: warn 2022-11-23T03:11:05.4853932Z path: test/**/*.json 2022-11-23T03:11:05.4854181Z env: 2022-11-23T03:11:05.4854431Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:11:05.4854754Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:11:05.4855270Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:11:05.4855820Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:11:05.4856184Z ##[endgroup] 2022-11-23T03:11:05.7447107Z With the provided path, there will be 3 files uploaded 2022-11-23T03:11:05.7449244Z Starting artifact upload 2022-11-23T03:11:05.7451545Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T03:11:05.7453141Z Artifact name is valid! 2022-11-23T03:11:05.8728699Z Container for artifact "test-jsons-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437735.zip" successfully created. Starting upload of file(s) 2022-11-23T03:11:06.2044170Z Total size of all the files uploaded is 29304 bytes 2022-11-23T03:11:06.2045253Z File upload process has finished. Finalizing the artifact upload 2022-11-23T03:11:06.3085107Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T03:11:06.3085768Z 2022-11-23T03:11:06.3086276Z The raw size of all the files that were specified for upload is 301646 bytes 2022-11-23T03:11:06.3087669Z The size of all the files that were uploaded is 29304 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T03:11:06.3088547Z 2022-11-23T03:11:06.3090132Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T03:11:06.3091201Z 2022-11-23T03:11:06.3092461Z Artifact test-jsons-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437735.zip has been successfully uploaded! 2022-11-23T03:11:06.3231463Z ##[group]Run actions/upload-artifact@v3 2022-11-23T03:11:06.3232056Z with: 2022-11-23T03:11:06.3232969Z name: test-reports-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437735.zip 2022-11-23T03:11:06.3233914Z retention-days: 14 2022-11-23T03:11:06.3234476Z if-no-files-found: ignore 2022-11-23T03:11:06.3235094Z path: test/**/*.xml test/**/*.csv 2022-11-23T03:11:06.3235633Z env: 2022-11-23T03:11:06.3236325Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:11:06.3237214Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:11:06.3238018Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:11:06.3238569Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:11:06.3238934Z ##[endgroup] 2022-11-23T03:11:06.6381166Z With the provided path, there will be 794 files uploaded 2022-11-23T03:11:06.6381770Z Starting artifact upload 2022-11-23T03:11:06.6382643Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T03:11:06.6383259Z Artifact name is valid! 2022-11-23T03:11:06.7698751Z Container for artifact "test-reports-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437735.zip" successfully created. Starting upload of file(s) 2022-11-23T03:11:16.7849786Z Total file count: 794 ---- Processed file #115 (14.4%) 2022-11-23T03:11:26.7851268Z Total file count: 794 ---- Processed file #232 (29.2%) 2022-11-23T03:11:36.7856795Z Total file count: 794 ---- Processed file #343 (43.1%) 2022-11-23T03:11:46.7857416Z Total file count: 794 ---- Processed file #456 (57.4%) 2022-11-23T03:11:56.7860868Z Total file count: 794 ---- Processed file #575 (72.4%) 2022-11-23T03:12:06.7864194Z Total file count: 794 ---- Processed file #681 (85.7%) 2022-11-23T03:12:16.6516791Z Total size of all the files uploaded is 245364 bytes 2022-11-23T03:12:16.6517899Z File upload process has finished. Finalizing the artifact upload 2022-11-23T03:12:16.9644395Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T03:12:16.9645047Z 2022-11-23T03:12:16.9645551Z The raw size of all the files that were specified for upload is 680470 bytes 2022-11-23T03:12:16.9648547Z The size of all the files that were uploaded is 245364 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T03:12:16.9649428Z 2022-11-23T03:12:16.9651037Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T03:12:16.9652821Z 2022-11-23T03:12:16.9654219Z Artifact test-reports-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437735.zip has been successfully uploaded! 2022-11-23T03:12:16.9881797Z ##[group]Run actions/upload-artifact@v3 2022-11-23T03:12:16.9882507Z with: 2022-11-23T03:12:16.9883548Z name: usage-log-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437735.zip 2022-11-23T03:12:16.9884652Z retention-days: 14 2022-11-23T03:12:16.9885318Z if-no-files-found: ignore 2022-11-23T03:12:16.9886007Z path: usage_log.txt 2022-11-23T03:12:16.9886598Z env: 2022-11-23T03:12:16.9887205Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:12:16.9888000Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:12:16.9889310Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:12:16.9890747Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:12:16.9891654Z ##[endgroup] 2022-11-23T03:12:17.0699045Z With the provided path, there will be 1 file uploaded 2022-11-23T03:12:17.0701133Z Starting artifact upload 2022-11-23T03:12:17.0703440Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T03:12:17.0705050Z Artifact name is valid! 2022-11-23T03:12:17.1979751Z Container for artifact "usage-log-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437735.zip" successfully created. Starting upload of file(s) 2022-11-23T03:12:17.7257594Z Total size of all the files uploaded is 344093 bytes 2022-11-23T03:12:17.7259164Z File upload process has finished. Finalizing the artifact upload 2022-11-23T03:12:17.8222600Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T03:12:17.8223276Z 2022-11-23T03:12:17.8223791Z The raw size of all the files that were specified for upload is 7024959 bytes 2022-11-23T03:12:17.8225198Z The size of all the files that were uploaded is 344093 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T03:12:17.8226072Z 2022-11-23T03:12:17.8227651Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T03:12:17.8228716Z 2022-11-23T03:12:17.8229952Z Artifact usage-log-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437735.zip has been successfully uploaded! 2022-11-23T03:12:17.8367704Z ##[group]Run set -x 2022-11-23T03:12:17.8368136Z set -x 2022-11-23T03:12:17.8368619Z python3 -m pip install -r requirements.txt 2022-11-23T03:12:17.8369172Z python3 -m pip install boto3==1.19.12 2022-11-23T03:12:17.8369787Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T03:12:17.8414061Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T03:12:17.8414534Z env: 2022-11-23T03:12:17.8414929Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:12:17.8415432Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:12:17.8416220Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:12:17.8417081Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:12:17.8417664Z AWS_DEFAULT_REGION: us-east-1 2022-11-23T03:12:17.8418056Z BRANCH: master 2022-11-23T03:12:17.8418503Z TEST_CONFIG: distributed 2022-11-23T03:12:17.8418878Z SHARD_NUMBER: 1 2022-11-23T03:12:17.8419353Z BUILD_ENVIRONMENT: linux-focal-rocm5.2-py3.8 2022-11-23T03:12:17.8419833Z PR_NUMBER: 2022-11-23T03:12:17.8420236Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T03:12:17.8420692Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T03:12:17.8421186Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T03:12:17.8421583Z TAG: 2022-11-23T03:12:17.8421936Z WORKFLOW_ID: 3528394938 2022-11-23T03:12:17.8422592Z GITHUB_TOKEN: *** 2022-11-23T03:12:17.8423068Z AWS_ACCESS_KEY_ID: *** 2022-11-23T03:12:17.8423665Z AWS_SECRET_ACCESS_KEY: *** 2022-11-23T03:12:17.8424101Z GHA_WORKFLOW_JOB_ID: 9655437735 2022-11-23T03:12:17.8424504Z ##[endgroup] 2022-11-23T03:12:17.8501809Z + python3 -m pip install -r requirements.txt 2022-11-23T03:12:18.6730640Z Collecting astunparse (from -r requirements.txt (line 2)) 2022-11-23T03:12:18.7340969Z Using cached https://files.pythonhosted.org/packages/2b/03/13dde6512ad7b4557eb792fbcf0c653af6076b81e5941d36ec61f7ce6028/astunparse-1.6.3-py2.py3-none-any.whl 2022-11-23T03:12:18.7397499Z Collecting expecttest (from -r requirements.txt (line 3)) 2022-11-23T03:12:18.7556762Z Using cached https://files.pythonhosted.org/packages/a6/26/1a287e44618c14659db0256bc1ee239c2134f9c863cb9a85813ecab73413/expecttest-0.1.4-py3-none-any.whl 2022-11-23T03:12:18.7575711Z Collecting future (from -r requirements.txt (line 4)) 2022-11-23T03:12:18.8128598Z Collecting hypothesis (from -r requirements.txt (line 5)) 2022-11-23T03:12:19.4577821Z Using cached https://files.pythonhosted.org/packages/82/7e/01691560a3a98bb1ae909affde49392f599eef3a0b91c27b992bbbde2abb/hypothesis-6.31.6-py3-none-any.whl 2022-11-23T03:12:19.5019045Z Collecting numpy (from -r requirements.txt (line 6)) 2022-11-23T03:12:20.0930209Z Using cached https://files.pythonhosted.org/packages/45/b2/6c7545bb7a38754d63048c7696804a0d947328125d81bf12beaa692c3ae3/numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl 2022-11-23T03:12:20.4073626Z Collecting psutil (from -r requirements.txt (line 7)) 2022-11-23T03:12:20.5935535Z Collecting pyyaml (from -r requirements.txt (line 8)) 2022-11-23T03:12:20.6600969Z Using cached https://files.pythonhosted.org/packages/b3/85/79b9e5b4e8d3c0ac657f4e8617713cca8408f6cdc65d2ee6554217cedff1/PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl 2022-11-23T03:12:20.6737854Z Collecting requests (from -r requirements.txt (line 9)) 2022-11-23T03:12:20.7164035Z Using cached https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl 2022-11-23T03:12:20.7292683Z Collecting setuptools (from -r requirements.txt (line 10)) 2022-11-23T03:12:21.0274357Z Using cached https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl 2022-11-23T03:12:21.0719417Z Collecting six (from -r requirements.txt (line 11)) 2022-11-23T03:12:21.0916862Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-11-23T03:12:21.0932003Z Collecting types-dataclasses (from -r requirements.txt (line 12)) 2022-11-23T03:12:21.1078756Z Using cached https://files.pythonhosted.org/packages/31/85/23ab2bbc280266af5bf22ded4e070946d1694d1721ced90666b649eaa795/types_dataclasses-0.6.6-py3-none-any.whl 2022-11-23T03:12:21.1090360Z Collecting typing_extensions (from -r requirements.txt (line 13)) 2022-11-23T03:12:21.1294672Z Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl 2022-11-23T03:12:21.1311041Z Collecting sympy (from -r requirements.txt (line 14)) 2022-11-23T03:12:21.2970873Z Using cached https://files.pythonhosted.org/packages/78/43/33c5a5e7fbafbf51520f4e09cb0634a1ca1d4cd5469c57967e43183d7a42/sympy-1.9-py3-none-any.whl 2022-11-23T03:12:21.5318200Z Collecting filelock (from -r requirements.txt (line 15)) 2022-11-23T03:12:21.5535788Z Using cached https://files.pythonhosted.org/packages/84/ce/8916d10ef537f3f3b046843255f9799504aa41862bfa87844b9bdc5361cd/filelock-3.4.1-py3-none-any.whl 2022-11-23T03:12:21.5610280Z Collecting networkx (from -r requirements.txt (line 16)) 2022-11-23T03:12:21.6194745Z Using cached https://files.pythonhosted.org/packages/f3/b7/c7f488101c0bb5e4178f3cde416004280fd40262433496830de8a8c21613/networkx-2.5.1-py3-none-any.whl 2022-11-23T03:12:21.6988139Z Collecting jinja2 (from -r requirements.txt (line 17)) 2022-11-23T03:12:21.7249242Z Using cached https://files.pythonhosted.org/packages/20/9a/e5d9ec41927401e41aea8af6d16e78b5e612bca4699d417f646a9610a076/Jinja2-3.0.3-py3-none-any.whl 2022-11-23T03:12:21.7318199Z Collecting wheel<1.0,>=0.23.0 (from astunparse->-r requirements.txt (line 2)) 2022-11-23T03:12:21.7684232Z Using cached https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl 2022-11-23T03:12:21.7725333Z Collecting attrs>=19.2.0 (from hypothesis->-r requirements.txt (line 5)) 2022-11-23T03:12:21.8238422Z Using cached https://files.pythonhosted.org/packages/f2/bc/d817287d1aa01878af07c19505fafd1165cd6a119e9d0821ca1d1c20312d/attrs-22.1.0-py2.py3-none-any.whl 2022-11-23T03:12:21.8475620Z Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis->-r requirements.txt (line 5)) 2022-11-23T03:12:21.8683957Z Using cached https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl 2022-11-23T03:12:21.8708458Z Collecting certifi>=2017.4.17 (from requests->-r requirements.txt (line 9)) 2022-11-23T03:12:21.8929204Z Using cached https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl 2022-11-23T03:12:21.8964148Z Collecting urllib3<1.27,>=1.21.1 (from requests->-r requirements.txt (line 9)) 2022-11-23T03:12:21.9346823Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T03:12:21.9529509Z Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests->-r requirements.txt (line 9)) 2022-11-23T03:12:22.0277642Z Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2022-11-23T03:12:22.0315512Z Collecting idna<4,>=2.5; python_version >= "3" (from requests->-r requirements.txt (line 9)) 2022-11-23T03:12:22.0455389Z Using cached https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl 2022-11-23T03:12:22.0487063Z Collecting mpmath>=0.19 (from sympy->-r requirements.txt (line 14)) 2022-11-23T03:12:22.0711937Z Using cached https://files.pythonhosted.org/packages/d4/cf/3965bddbb4f1a61c49aacae0e78fd1fe36b5dc36c797b31f30cf07dcbbb7/mpmath-1.2.1-py3-none-any.whl 2022-11-23T03:12:22.0925325Z Collecting decorator<5,>=4.3 (from networkx->-r requirements.txt (line 16)) 2022-11-23T03:12:22.1162970Z Using cached https://files.pythonhosted.org/packages/ed/1b/72a1821152d07cf1d8b6fce298aeb06a7eb90f4d6d41acec9861e7cc6df0/decorator-4.4.2-py2.py3-none-any.whl 2022-11-23T03:12:22.1178313Z Collecting MarkupSafe>=2.0 (from jinja2->-r requirements.txt (line 17)) 2022-11-23T03:12:22.1849351Z Using cached https://files.pythonhosted.org/packages/fc/d6/57f9a97e56447a1e340f8574836d3b636e2c14de304943836bd645fa9c7e/MarkupSafe-2.0.1-cp36-cp36m-manylinux1_x86_64.whl 2022-11-23T03:12:22.1875042Z Installing collected packages: six, wheel, astunparse, expecttest, future, attrs, sortedcontainers, hypothesis, numpy, psutil, pyyaml, certifi, urllib3, charset-normalizer, idna, requests, setuptools, types-dataclasses, typing-extensions, mpmath, sympy, filelock, decorator, networkx, MarkupSafe, jinja2 2022-11-23T03:12:30.7560956Z Successfully installed MarkupSafe-2.0.1 astunparse-1.6.3 attrs-22.1.0 certifi-2022.9.24 charset-normalizer-2.0.12 decorator-4.4.2 expecttest-0.1.4 filelock-3.4.1 future-0.18.2 hypothesis-6.31.6 idna-3.4 jinja2-3.0.3 mpmath-1.2.1 networkx-2.5.1 numpy-1.19.5 psutil-5.9.4 pyyaml-6.0 requests-2.27.1 setuptools-59.6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.9 types-dataclasses-0.6.6 typing-extensions-4.1.1 urllib3-1.26.12 wheel-0.37.1 2022-11-23T03:12:30.8989545Z + python3 -m pip install boto3==1.19.12 2022-11-23T03:12:31.7048517Z Collecting boto3==1.19.12 2022-11-23T03:12:32.2612790Z Using cached https://files.pythonhosted.org/packages/5e/e1/156846b09fca21b9b164c54200011e3bd17f29187cbfc6903a8e0281a304/boto3-1.19.12-py3-none-any.whl 2022-11-23T03:12:32.2735208Z Collecting s3transfer<0.6.0,>=0.5.0 (from boto3==1.19.12) 2022-11-23T03:12:32.2958695Z Using cached https://files.pythonhosted.org/packages/7b/9c/f51775ebe7df5a7aa4e7c79ed671bde94e154bd968aca8d65bb24aba0c8c/s3transfer-0.5.2-py3-none-any.whl 2022-11-23T03:12:32.3012690Z Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.19.12) 2022-11-23T03:12:32.3232881Z Using cached https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl 2022-11-23T03:12:32.3267699Z Collecting botocore<1.23.0,>=1.22.12 (from boto3==1.19.12) 2022-11-23T03:12:33.0003390Z Using cached https://files.pythonhosted.org/packages/6a/73/552b27e3a1b4f83630907c4958be78e9d4c906e73efd554ebd5e21cb1692/botocore-1.22.12-py3-none-any.whl 2022-11-23T03:12:33.2922575Z Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T03:12:33.3206836Z Using cached https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl 2022-11-23T03:12:33.3271807Z Collecting urllib3<1.27,>=1.25.4 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T03:12:33.3649053Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T03:12:33.3823607Z Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T03:12:33.3957654Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-11-23T03:12:33.3974629Z Installing collected packages: six, python-dateutil, jmespath, urllib3, botocore, s3transfer, boto3 2022-11-23T03:12:33.8955808Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.12 2022-11-23T03:12:33.9815443Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T03:12:43.9066551Z [scribe] Scribe access token not provided, sending report via boto3... 2022-11-23T03:12:43.9070910Z 2022-11-23T03:12:43.9072896Z ----- Historic stats comparison result ------ 2022-11-23T03:12:43.9073493Z 2022-11-23T03:12:43.9074049Z job: linux-focal-rocm5.2-py3.8 2022-11-23T03:12:43.9078611Z commit: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T03:12:43.9079162Z 2022-11-23T03:12:43.9079692Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-11-23T03:12:43.9080326Z 2022-11-23T03:12:43.9080551Z : (master) 2022-11-23T03:12:43.9084699Z | 2022-11-23T03:12:43.9085467Z * 1cfd3858ac (HEAD) total time 1750.07s 2022-11-23T03:12:43.9098087Z * 26322544b8 (base) 2 reports, total time 10567.84s � 1421.60s 2022-11-23T03:12:43.9099392Z * 7f4b4d2827 2 reports, total time 8461.32s � 1168.97s 2022-11-23T03:12:43.9100568Z * b50699f247 2 reports, total time 9450.96s � 2404.87s 2022-11-23T03:12:43.9101724Z * 8bf8e4d71e 2 reports, total time 9492.36s � 2525.06s 2022-11-23T03:12:43.9102923Z * ce856cee7e 2 reports, total time 10405.71s � 1187.54s 2022-11-23T03:12:43.9104060Z * 391b593ca2 2 reports, total time 10409.50s � 1113.28s 2022-11-23T03:12:43.9105175Z * 5bba783d21 2 reports, total time 10442.70s � 1177.08s 2022-11-23T03:12:43.9106335Z * ea920a1115 2 reports, total time 8384.52s � 1109.09s 2022-11-23T03:12:43.9107480Z * 74e62a1fef 2 reports, total time 8516.95s � 1120.62s 2022-11-23T03:12:43.9108613Z * 00b7d8ef23 2 reports, total time 10408.58s � 1157.84s 2022-11-23T03:12:43.9109291Z | 2022-11-23T03:12:43.9109799Z : 2022-11-23T03:12:43.9110092Z 2022-11-23T03:12:43.9110505Z Removed (across 518 suites) 0 tests, totaling 0.00s 2022-11-23T03:12:43.9111420Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-11-23T03:12:43.9112339Z Added (across 69 suites) 962 tests, totaling +2512.20s 2022-11-23T03:12:43.9600911Z ##[group]Run # Only stop the docker container we started since there might be multiple runners on this host. 2022-11-23T03:12:43.9602332Z # Only stop the docker container we started since there might be multiple runners on this host. 2022-11-23T03:12:43.9603611Z docker stop "de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1" || true 2022-11-23T03:12:43.9604676Z # Prune all of the docker containers. 2022-11-23T03:12:43.9605645Z # Might fail if a prune is already in progress by another runner. 2022-11-23T03:12:43.9606571Z docker container prune -f || true 2022-11-23T03:12:43.9607498Z # Prune everything docker if there are more than 10 images (~200GB). 2022-11-23T03:12:43.9608513Z # This is easier than using a time filter, e.g., "until=24h". 2022-11-23T03:12:43.9609525Z # Might fail if a prune is already in progress by another runner. 2022-11-23T03:12:43.9610482Z image_count=$(docker images | wc -l) 2022-11-23T03:12:43.9611262Z if [[ ${image_count} -gt 10 ]]; then 2022-11-23T03:12:43.9612034Z  echo "Purging all docker caches" 2022-11-23T03:12:43.9612824Z  docker system prune -af || true 2022-11-23T03:12:43.9613488Z else 2022-11-23T03:12:43.9614468Z  echo "Will not purge docker, only ${image_count} images found" 2022-11-23T03:12:43.9615226Z fi 2022-11-23T03:12:43.9678841Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T03:12:43.9679568Z env: 2022-11-23T03:12:43.9680174Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:12:43.9680950Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:12:43.9681455Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:12:43.9682008Z CONTAINER_NAME: de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:12:43.9682369Z ##[endgroup] 2022-11-23T03:12:44.4913594Z de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:12:55.7772997Z Deleted Containers: 2022-11-23T03:12:55.7773666Z de3c05a7b1351905e3fe55974c4315adc238b9fd2809a2d40a3bf3e8b3ad8cc1 2022-11-23T03:12:55.7774062Z 2022-11-23T03:12:55.7774297Z Total reclaimed space: 8.468GB 2022-11-23T03:12:55.8384127Z Will not purge docker, only 3 images found 2022-11-23T03:12:55.8477690Z Post job cleanup. 2022-11-23T03:12:55.8533202Z Post job cleanup. 2022-11-23T03:12:55.9921515Z [command]/usr/bin/git version 2022-11-23T03:12:55.9988016Z git version 2.35.1 2022-11-23T03:12:56.0043496Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/ac7a97a9-7fe4-49ac-a4b5-84120eac50f0' before making global git config changes 2022-11-23T03:12:56.0045079Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T03:12:56.0047964Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T03:12:56.0122487Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T03:12:56.0184037Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T03:12:56.0655684Z Entering 'android/libs/fbjni' 2022-11-23T03:12:56.0723839Z Entering 'third_party/FP16' 2022-11-23T03:12:56.0807385Z Entering 'third_party/FXdiv' 2022-11-23T03:12:56.0885005Z Entering 'third_party/NNPACK' 2022-11-23T03:12:56.0961396Z Entering 'third_party/QNNPACK' 2022-11-23T03:12:56.1040805Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T03:12:56.1115222Z Entering 'third_party/XNNPACK' 2022-11-23T03:12:56.1233698Z Entering 'third_party/benchmark' 2022-11-23T03:12:56.1303683Z Entering 'third_party/cpuinfo' 2022-11-23T03:12:56.1379463Z Entering 'third_party/cub' 2022-11-23T03:12:56.1455751Z Entering 'third_party/cudnn_frontend' 2022-11-23T03:12:56.1546874Z Entering 'third_party/cutlass' 2022-11-23T03:12:56.1642395Z Entering 'third_party/eigen' 2022-11-23T03:12:56.1733235Z Entering 'third_party/fbgemm' 2022-11-23T03:12:56.1816845Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T03:12:56.1894885Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T03:12:56.1981461Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T03:12:56.2061700Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T03:12:56.2146285Z Entering 'third_party/flatbuffers' 2022-11-23T03:12:56.2240262Z Entering 'third_party/fmt' 2022-11-23T03:12:56.2327418Z Entering 'third_party/foxi' 2022-11-23T03:12:56.2411464Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T03:12:56.2496026Z Entering 'third_party/gloo' 2022-11-23T03:12:56.2584950Z Entering 'third_party/googletest' 2022-11-23T03:12:56.2675006Z Entering 'third_party/ideep' 2022-11-23T03:12:56.2760024Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T03:12:56.2851345Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T03:12:56.2958897Z Entering 'third_party/ios-cmake' 2022-11-23T03:12:56.3038222Z Entering 'third_party/ittapi' 2022-11-23T03:12:56.3116932Z Entering 'third_party/kineto' 2022-11-23T03:12:56.3201140Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T03:12:56.3282479Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T03:12:56.3372668Z Entering 'third_party/nccl/nccl' 2022-11-23T03:12:56.3460059Z Entering 'third_party/neon2sse' 2022-11-23T03:12:56.3547386Z Entering 'third_party/nlohmann' 2022-11-23T03:12:56.3641455Z Entering 'third_party/onnx' 2022-11-23T03:12:56.3766949Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T03:12:56.3850408Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T03:12:56.3937980Z Entering 'third_party/onnx-tensorrt' 2022-11-23T03:12:56.4016717Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T03:12:56.4114966Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T03:12:56.4197566Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T03:12:56.4281279Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T03:12:56.4378253Z Entering 'third_party/pocketfft' 2022-11-23T03:12:56.4464552Z Entering 'third_party/protobuf' 2022-11-23T03:12:56.4561022Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T03:12:56.4643011Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T03:12:56.4731879Z Entering 'third_party/psimd' 2022-11-23T03:12:56.4818296Z Entering 'third_party/pthreadpool' 2022-11-23T03:12:56.4901749Z Entering 'third_party/pybind11' 2022-11-23T03:12:56.4988011Z Entering 'third_party/python-enum' 2022-11-23T03:12:56.5077043Z Entering 'third_party/python-peachpy' 2022-11-23T03:12:56.5164871Z Entering 'third_party/python-six' 2022-11-23T03:12:56.5248678Z Entering 'third_party/sleef' 2022-11-23T03:12:56.5337021Z Entering 'third_party/tbb' 2022-11-23T03:12:56.5431273Z Entering 'third_party/tensorpipe' 2022-11-23T03:12:56.5511072Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T03:12:56.5594777Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T03:12:56.5672744Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T03:12:56.5759630Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T03:12:56.5841924Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T03:12:56.5920505Z Entering 'third_party/zstd' 2022-11-23T03:12:56.6042503Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T03:12:56.6111999Z http.https://github.com/.extraheader 2022-11-23T03:12:56.6134072Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-11-23T03:12:56.6214799Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T03:12:56.6739908Z Entering 'android/libs/fbjni' 2022-11-23T03:12:56.6785586Z http.https://github.com/.extraheader 2022-11-23T03:12:56.6857977Z Entering 'third_party/FP16' 2022-11-23T03:12:56.6899595Z http.https://github.com/.extraheader 2022-11-23T03:12:56.6969030Z Entering 'third_party/FXdiv' 2022-11-23T03:12:56.7014811Z http.https://github.com/.extraheader 2022-11-23T03:12:56.7079650Z Entering 'third_party/NNPACK' 2022-11-23T03:12:56.7124125Z http.https://github.com/.extraheader 2022-11-23T03:12:56.7190536Z Entering 'third_party/QNNPACK' 2022-11-23T03:12:56.7226227Z http.https://github.com/.extraheader 2022-11-23T03:12:56.7298045Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T03:12:56.7337381Z http.https://github.com/.extraheader 2022-11-23T03:12:56.7404453Z Entering 'third_party/XNNPACK' 2022-11-23T03:12:56.7450587Z http.https://github.com/.extraheader 2022-11-23T03:12:56.7547073Z Entering 'third_party/benchmark' 2022-11-23T03:12:56.7589317Z http.https://github.com/.extraheader 2022-11-23T03:12:56.7660673Z Entering 'third_party/cpuinfo' 2022-11-23T03:12:56.7705178Z http.https://github.com/.extraheader 2022-11-23T03:12:56.7774668Z Entering 'third_party/cub' 2022-11-23T03:12:56.7812506Z http.https://github.com/.extraheader 2022-11-23T03:12:56.7883368Z Entering 'third_party/cudnn_frontend' 2022-11-23T03:12:56.7929940Z http.https://github.com/.extraheader 2022-11-23T03:12:56.8016183Z Entering 'third_party/cutlass' 2022-11-23T03:12:56.8058172Z http.https://github.com/.extraheader 2022-11-23T03:12:56.8144784Z Entering 'third_party/eigen' 2022-11-23T03:12:56.8189925Z http.https://github.com/.extraheader 2022-11-23T03:12:56.8270388Z Entering 'third_party/fbgemm' 2022-11-23T03:12:56.8307369Z http.https://github.com/.extraheader 2022-11-23T03:12:56.8376172Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T03:12:56.8421383Z http.https://github.com/.extraheader 2022-11-23T03:12:56.8498888Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T03:12:56.8545127Z http.https://github.com/.extraheader 2022-11-23T03:12:56.8623382Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T03:12:56.8667953Z http.https://github.com/.extraheader 2022-11-23T03:12:56.8741116Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T03:12:56.8786999Z http.https://github.com/.extraheader 2022-11-23T03:12:56.8864327Z Entering 'third_party/flatbuffers' 2022-11-23T03:12:56.8909856Z http.https://github.com/.extraheader 2022-11-23T03:12:56.8986735Z Entering 'third_party/fmt' 2022-11-23T03:12:56.9028334Z http.https://github.com/.extraheader 2022-11-23T03:12:56.9097956Z Entering 'third_party/foxi' 2022-11-23T03:12:56.9141760Z http.https://github.com/.extraheader 2022-11-23T03:12:56.9212001Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T03:12:56.9258287Z http.https://github.com/.extraheader 2022-11-23T03:12:56.9334261Z Entering 'third_party/gloo' 2022-11-23T03:12:56.9379260Z http.https://github.com/.extraheader 2022-11-23T03:12:56.9450824Z Entering 'third_party/googletest' 2022-11-23T03:12:56.9493847Z http.https://github.com/.extraheader 2022-11-23T03:12:56.9570133Z Entering 'third_party/ideep' 2022-11-23T03:12:56.9613588Z http.https://github.com/.extraheader 2022-11-23T03:12:56.9683178Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T03:12:56.9728834Z http.https://github.com/.extraheader 2022-11-23T03:12:56.9810118Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T03:12:56.9858018Z http.https://github.com/.extraheader 2022-11-23T03:12:56.9956033Z Entering 'third_party/ios-cmake' 2022-11-23T03:12:57.0002094Z http.https://github.com/.extraheader 2022-11-23T03:12:57.0076370Z Entering 'third_party/ittapi' 2022-11-23T03:12:57.0121188Z http.https://github.com/.extraheader 2022-11-23T03:12:57.0192593Z Entering 'third_party/kineto' 2022-11-23T03:12:57.0238917Z http.https://github.com/.extraheader 2022-11-23T03:12:57.0313760Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T03:12:57.0358329Z http.https://github.com/.extraheader 2022-11-23T03:12:57.0429632Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T03:12:57.0470875Z http.https://github.com/.extraheader 2022-11-23T03:12:57.0547397Z Entering 'third_party/nccl/nccl' 2022-11-23T03:12:57.0592979Z http.https://github.com/.extraheader 2022-11-23T03:12:57.0669652Z Entering 'third_party/neon2sse' 2022-11-23T03:12:57.0715216Z http.https://github.com/.extraheader 2022-11-23T03:12:57.0783992Z Entering 'third_party/nlohmann' 2022-11-23T03:12:57.0827221Z http.https://github.com/.extraheader 2022-11-23T03:12:57.0907656Z Entering 'third_party/onnx' 2022-11-23T03:12:57.0953255Z http.https://github.com/.extraheader 2022-11-23T03:12:57.1048432Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T03:12:57.1092049Z http.https://github.com/.extraheader 2022-11-23T03:12:57.1163575Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T03:12:57.1208166Z http.https://github.com/.extraheader 2022-11-23T03:12:57.1288445Z Entering 'third_party/onnx-tensorrt' 2022-11-23T03:12:57.1333241Z http.https://github.com/.extraheader 2022-11-23T03:12:57.1404851Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T03:12:57.1448342Z http.https://github.com/.extraheader 2022-11-23T03:12:57.1535804Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T03:12:57.1578195Z http.https://github.com/.extraheader 2022-11-23T03:12:57.1654412Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T03:12:57.1700194Z http.https://github.com/.extraheader 2022-11-23T03:12:57.1773502Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T03:12:57.1820570Z http.https://github.com/.extraheader 2022-11-23T03:12:57.1902673Z Entering 'third_party/pocketfft' 2022-11-23T03:12:57.1948653Z http.https://github.com/.extraheader 2022-11-23T03:12:57.2025252Z Entering 'third_party/protobuf' 2022-11-23T03:12:57.2068810Z http.https://github.com/.extraheader 2022-11-23T03:12:57.2145946Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T03:12:57.2189828Z http.https://github.com/.extraheader 2022-11-23T03:12:57.2258599Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T03:12:57.2300705Z http.https://github.com/.extraheader 2022-11-23T03:12:57.2377027Z Entering 'third_party/psimd' 2022-11-23T03:12:57.2423283Z http.https://github.com/.extraheader 2022-11-23T03:12:57.2493930Z Entering 'third_party/pthreadpool' 2022-11-23T03:12:57.2539535Z http.https://github.com/.extraheader 2022-11-23T03:12:57.2605277Z Entering 'third_party/pybind11' 2022-11-23T03:12:57.2650922Z http.https://github.com/.extraheader 2022-11-23T03:12:57.2727259Z Entering 'third_party/python-enum' 2022-11-23T03:12:57.2772508Z http.https://github.com/.extraheader 2022-11-23T03:12:57.2848364Z Entering 'third_party/python-peachpy' 2022-11-23T03:12:57.2894235Z http.https://github.com/.extraheader 2022-11-23T03:12:57.2967717Z Entering 'third_party/python-six' 2022-11-23T03:12:57.3012510Z http.https://github.com/.extraheader 2022-11-23T03:12:57.3088272Z Entering 'third_party/sleef' 2022-11-23T03:12:57.3133906Z http.https://github.com/.extraheader 2022-11-23T03:12:57.3209794Z Entering 'third_party/tbb' 2022-11-23T03:12:57.3255401Z http.https://github.com/.extraheader 2022-11-23T03:12:57.3331249Z Entering 'third_party/tensorpipe' 2022-11-23T03:12:57.3377275Z http.https://github.com/.extraheader 2022-11-23T03:12:57.3452309Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T03:12:57.3495239Z http.https://github.com/.extraheader 2022-11-23T03:12:57.3566027Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T03:12:57.3607289Z http.https://github.com/.extraheader 2022-11-23T03:12:57.3678804Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T03:12:57.3722488Z http.https://github.com/.extraheader 2022-11-23T03:12:57.3799050Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T03:12:57.3841859Z http.https://github.com/.extraheader 2022-11-23T03:12:57.3910239Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T03:12:57.3954785Z http.https://github.com/.extraheader 2022-11-23T03:12:57.4039173Z Entering 'third_party/zstd' 2022-11-23T03:12:57.4081063Z http.https://github.com/.extraheader 2022-11-23T03:12:57.4596366Z Cleaning up orphan processes