2022-11-23T01:30:12.2029846Z Requested labels: linux.rocm.gpu 2022-11-23T01:30:12.2029942Z Job defined at: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/master 2022-11-23T01:30:12.2029965Z Waiting for a runner to pick up this job... 2022-11-23T01:30:12.3945561Z Job is about to start running on the runner: worker-rocm-amd-106 (repository) 2022-11-23T01:30:15.5035990Z Current runner version: '2.299.1' 2022-11-23T01:30:15.5042741Z Runner name: 'worker-rocm-amd-106' 2022-11-23T01:30:15.5043257Z Runner group name: 'Default' 2022-11-23T01:30:15.5044004Z Machine name: 'jenkins-worker-rocm-amd-106' 2022-11-23T01:30:15.5046315Z ##[group]GITHUB_TOKEN Permissions 2022-11-23T01:30:15.5046913Z Actions: write 2022-11-23T01:30:15.5047215Z Checks: write 2022-11-23T01:30:15.5047501Z Contents: write 2022-11-23T01:30:15.5047821Z Deployments: write 2022-11-23T01:30:15.5048141Z Discussions: write 2022-11-23T01:30:15.5048444Z Issues: write 2022-11-23T01:30:15.5048739Z Metadata: read 2022-11-23T01:30:15.5049024Z Packages: write 2022-11-23T01:30:15.5049320Z Pages: write 2022-11-23T01:30:15.5049623Z PullRequests: write 2022-11-23T01:30:15.5049971Z RepositoryProjects: write 2022-11-23T01:30:15.5050325Z SecurityEvents: write 2022-11-23T01:30:15.5050625Z Statuses: write 2022-11-23T01:30:15.5050920Z ##[endgroup] 2022-11-23T01:30:15.5055208Z Secret source: Actions 2022-11-23T01:30:15.5055843Z Prepare workflow directory 2022-11-23T01:30:15.8939360Z Prepare all required actions 2022-11-23T01:30:15.9157873Z Getting action download info 2022-11-23T01:30:16.1987871Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f) 2022-11-23T01:30:22.8113539Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd) 2022-11-23T01:30:24.0360531Z Getting action download info 2022-11-23T01:30:24.2623848Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076) 2022-11-23T01:30:25.2922853Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml 2022-11-23T01:30:25.2925320Z ##[group] Inputs 2022-11-23T01:30:25.2925739Z build-environment: linux-focal-rocm5.2-py3.8 2022-11-23T01:30:25.2926326Z test-matrix: { include: [ { config: "distributed", shard: 1, num_shards: 2, runner: "linux.rocm.gpu" }, { config: "distributed", shard: 2, num_shards: 2, runner: "linux.rocm.gpu" }, ]} 2022-11-23T01:30:25.2927068Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:30:25.2927532Z sync-tag: 2022-11-23T01:30:25.2927794Z ##[endgroup] 2022-11-23T01:30:25.2928531Z Complete job name: linux-focal-rocm5.2-py3.8-distributed / test (distributed, 2, 2, linux.rocm.gpu, rerun_disabled_tests) 2022-11-23T01:30:25.4310785Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2022-11-23T01:30:25.4311133Z with: 2022-11-23T01:30:25.4311340Z no-sudo: true 2022-11-23T01:30:25.4311567Z submodules: recursive 2022-11-23T01:30:25.4311786Z fetch-depth: 0 2022-11-23T01:30:25.4311965Z env: 2022-11-23T01:30:25.4312172Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:25.4312396Z ##[endgroup] 2022-11-23T01:30:25.4524215Z ##[group]Run retry () { 2022-11-23T01:30:25.4524484Z retry () { 2022-11-23T01:30:25.4524732Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2022-11-23T01:30:25.4524959Z } 2022-11-23T01:30:25.4525165Z echo "${GITHUB_WORKSPACE}" 2022-11-23T01:30:25.4525406Z if [ -z "${NO_SUDO}" ]; then 2022-11-23T01:30:25.4525649Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T01:30:25.4525888Z else 2022-11-23T01:30:25.4526150Z  retry rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T01:30:25.4526391Z fi 2022-11-23T01:30:25.4526578Z mkdir "${GITHUB_WORKSPACE}" 2022-11-23T01:30:25.4555130Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:30:25.4555390Z env: 2022-11-23T01:30:25.4555587Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:25.4555774Z NO_SUDO: true 2022-11-23T01:30:25.4555966Z ##[endgroup] 2022-11-23T01:30:25.4755076Z /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:27.1501407Z ##[group]Run malfet/checkout@silent-checkout 2022-11-23T01:30:27.1501823Z with: 2022-11-23T01:30:27.1502163Z ref: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:30:27.1502719Z fetch-depth: 0 2022-11-23T01:30:27.1502996Z submodules: recursive 2022-11-23T01:30:27.1503325Z quiet-checkout: true 2022-11-23T01:30:27.1503671Z repository: pytorch/pytorch 2022-11-23T01:30:27.1504195Z token: *** 2022-11-23T01:30:27.1504510Z ssh-strict: true 2022-11-23T01:30:27.1504846Z persist-credentials: true 2022-11-23T01:30:27.1505167Z clean: true 2022-11-23T01:30:27.1505460Z lfs: false 2022-11-23T01:30:27.1505782Z set-safe-directory: true 2022-11-23T01:30:27.1506100Z env: 2022-11-23T01:30:27.1506399Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:27.1506676Z ##[endgroup] 2022-11-23T01:30:27.3222240Z Syncing repository: pytorch/pytorch 2022-11-23T01:30:27.3224180Z ##[group]Getting Git version info 2022-11-23T01:30:27.3224845Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-11-23T01:30:27.3225603Z [command]/usr/bin/git version 2022-11-23T01:30:27.3225948Z git version 2.37.1 2022-11-23T01:30:27.3226859Z ##[endgroup] 2022-11-23T01:30:27.3241602Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/67b8d739-3096-48d2-9988-86c739d85c3a' before making global git config changes 2022-11-23T01:30:27.3242259Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T01:30:27.3243013Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:27.3243728Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-11-23T01:30:27.3244541Z ##[group]Initializing the repository 2022-11-23T01:30:27.3245125Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:27.3294530Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-11-23T01:30:27.3295093Z hint: is subject to change. To configure the initial branch name to use in all 2022-11-23T01:30:27.3295574Z hint: of your new repositories, which will suppress this warning, call: 2022-11-23T01:30:27.3296011Z hint: 2022-11-23T01:30:27.3296509Z hint: git config --global init.defaultBranch 2022-11-23T01:30:27.3296880Z hint: 2022-11-23T01:30:27.3297367Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-11-23T01:30:27.3297974Z hint: 'development'. The just-created branch can be renamed via this command: 2022-11-23T01:30:27.3298544Z hint: 2022-11-23T01:30:27.3298926Z hint: git branch -m 2022-11-23T01:30:27.3303001Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2022-11-23T01:30:27.3312640Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-11-23T01:30:27.3371250Z ##[endgroup] 2022-11-23T01:30:27.3371895Z ##[group]Disabling automatic garbage collection 2022-11-23T01:30:27.3375044Z [command]/usr/bin/git config --local gc.auto 0 2022-11-23T01:30:27.3406937Z ##[endgroup] 2022-11-23T01:30:27.3407483Z ##[group]Setting up auth 2022-11-23T01:30:27.3414998Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T01:30:27.3462864Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T01:30:27.3876669Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T01:30:27.3931765Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T01:30:27.4387627Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T01:30:27.4470236Z ##[endgroup] 2022-11-23T01:30:27.4471775Z ##[group]Fetching the repository 2022-11-23T01:30:27.4482499Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-11-23T01:31:47.9391516Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object} 2022-11-23T01:31:47.9457860Z 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:31:47.9472012Z ##[endgroup] 2022-11-23T01:31:47.9473578Z ##[group]Determining the checkout info 2022-11-23T01:31:47.9475024Z ##[endgroup] 2022-11-23T01:31:47.9476377Z ##[group]Checking out the ref 2022-11-23T01:31:47.9484678Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:31:49.2288349Z ##[endgroup] 2022-11-23T01:31:49.2290013Z ##[group]Setting up auth for fetching submodules 2022-11-23T01:31:49.2303451Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T01:31:49.2376143Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-11-23T01:31:49.2418968Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2022-11-23T01:31:49.2471496Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2022-11-23T01:31:49.2506724Z ##[endgroup] 2022-11-23T01:31:49.2508206Z ##[group]Fetching submodules 2022-11-23T01:31:49.2510227Z [command]/usr/bin/git submodule sync --recursive 2022-11-23T01:31:49.3001291Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-11-23T01:31:49.3491663Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-11-23T01:31:49.3494724Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-11-23T01:31:49.3503441Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-11-23T01:31:49.3510866Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-11-23T01:31:49.3518871Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-11-23T01:31:49.3525365Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2022-11-23T01:31:49.3528357Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-11-23T01:31:49.3531355Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-11-23T01:31:49.3534891Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-11-23T01:31:49.3538773Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-11-23T01:31:49.3542959Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-11-23T01:31:49.3546478Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2022-11-23T01:31:49.3550627Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2022-11-23T01:31:49.3554790Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-11-23T01:31:49.3558714Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-11-23T01:31:49.3563434Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-11-23T01:31:49.3568011Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-11-23T01:31:49.3572793Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:31:49.3577497Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-11-23T01:31:49.3582711Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-11-23T01:31:49.3587582Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-11-23T01:31:49.3593031Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-11-23T01:31:49.3598244Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2022-11-23T01:31:49.3603900Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-11-23T01:31:49.3610790Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-11-23T01:31:49.3616733Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-11-23T01:31:49.3621444Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2022-11-23T01:31:49.3627045Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-11-23T01:31:49.3633342Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-11-23T01:31:49.3639467Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-11-23T01:31:49.3646152Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-11-23T01:31:49.3653619Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-11-23T01:31:49.3659193Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-11-23T01:31:49.3665748Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-11-23T01:31:49.3672766Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-11-23T01:31:49.3679511Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-11-23T01:31:49.3687540Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-11-23T01:31:49.3695282Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-11-23T01:31:49.3702142Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-11-23T01:31:49.3709376Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-11-23T01:31:49.3717482Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-11-23T01:31:49.3784863Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-11-23T01:31:50.5795260Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-11-23T01:31:51.5010692Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-11-23T01:31:52.3086940Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-11-23T01:31:53.5196531Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-11-23T01:31:54.6387631Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2022-11-23T01:31:57.8902627Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-11-23T01:32:04.9075914Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-11-23T01:32:06.3022458Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-11-23T01:32:08.1284565Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-11-23T01:32:10.8960452Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-11-23T01:32:18.3871529Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2022-11-23T01:32:21.2783127Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-11-23T01:32:33.0786227Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-11-23T01:32:34.8247218Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-11-23T01:32:37.4393878Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-11-23T01:32:39.7154961Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-11-23T01:32:40.5575169Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-11-23T01:32:43.9004514Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-11-23T01:32:45.0760810Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-11-23T01:32:47.0050186Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-11-23T01:32:48.3996698Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-11-23T01:32:49.2539249Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2022-11-23T01:32:50.3067242Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-11-23T01:32:53.1507379Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-11-23T01:32:54.5501628Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-11-23T01:32:55.7863148Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2022-11-23T01:33:03.2994272Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-11-23T01:33:06.0044222Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-11-23T01:33:07.3872754Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-11-23T01:33:08.3173878Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-11-23T01:33:16.0796389Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-11-23T01:33:16.8588612Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-11-23T01:33:17.9081784Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-11-23T01:33:19.6751217Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-11-23T01:33:20.5199827Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-11-23T01:33:21.7715852Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-11-23T01:33:23.0596713Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-11-23T01:33:24.6364779Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-11-23T01:33:30.3049911Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-11-23T01:33:32.1459886Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-11-23T01:33:35.8089982Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-11-23T01:33:35.8340611Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-11-23T01:33:35.8523692Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-11-23T01:33:35.8889850Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-11-23T01:33:35.9257700Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-11-23T01:33:35.9814184Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2022-11-23T01:33:36.6762644Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-11-23T01:33:36.7093433Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:33:36.8436729Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c' 2022-11-23T01:33:36.8933484Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-11-23T01:33:37.2078267Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3' 2022-11-23T01:33:37.6310808Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a' 2022-11-23T01:33:37.9085772Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2022-11-23T01:33:37.9723945Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a' 2022-11-23T01:33:37.9764257Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:37.9769112Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:37.9774304Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:37.9780315Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:37.9825859Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-11-23T01:33:40.1051358Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-11-23T01:33:41.9705715Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-11-23T01:33:43.9818586Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2022-11-23T01:33:45.0308411Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2022-11-23T01:33:45.1714295Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-11-23T01:33:45.2658288Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-11-23T01:33:45.2852299Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2022-11-23T01:33:45.4165504Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-11-23T01:33:45.4757937Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b' 2022-11-23T01:33:45.4937896Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-11-23T01:33:45.5565447Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-11-23T01:33:45.5971780Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195' 2022-11-23T01:33:45.6684826Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-11-23T01:33:45.6912371Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f' 2022-11-23T01:33:45.6941412Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:45.6987517Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-11-23T01:33:57.0692720Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d' 2022-11-23T01:33:57.0741679Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:57.0803403Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-11-23T01:34:09.1979427Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289' 2022-11-23T01:34:09.2265576Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-11-23T01:34:09.2648300Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2022-11-23T01:34:09.4055662Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562' 2022-11-23T01:34:09.4107770Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:09.4117208Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:09.4181671Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-11-23T01:34:12.2618617Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-11-23T01:34:14.8350209Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-11-23T01:34:14.9180585Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-11-23T01:34:14.9633883Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52' 2022-11-23T01:34:14.9911624Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-11-23T01:34:15.1339431Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2022-11-23T01:34:15.3887398Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b' 2022-11-23T01:34:15.3964250Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:15.3970367Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:15.4021447Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-11-23T01:34:16.7798815Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-11-23T01:34:19.5537994Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:34:19.6011364Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8' 2022-11-23T01:34:19.6304013Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-11-23T01:34:19.6339822Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:19.6399556Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-11-23T01:34:22.5765661Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-11-23T01:34:22.5823322Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:22.5828426Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:22.5894939Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-11-23T01:34:23.9822328Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-11-23T01:34:25.8434006Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-11-23T01:34:25.9449485Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-11-23T01:34:25.9489059Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:25.9531431Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-11-23T01:34:26.9112561Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:34:26.9321667Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-11-23T01:34:27.2689553Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-11-23T01:34:27.2743706Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:27.2748076Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:27.2802059Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-11-23T01:34:28.6166894Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-11-23T01:34:31.1598025Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-11-23T01:34:31.2621289Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-11-23T01:34:31.2813248Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-11-23T01:34:31.3054259Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-11-23T01:34:31.3589142Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917' 2022-11-23T01:34:31.3796495Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-11-23T01:34:31.4304228Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2022-11-23T01:34:31.4497012Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-11-23T01:34:31.5166883Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-11-23T01:34:31.6613777Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-11-23T01:34:31.7099274Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-11-23T01:34:31.7140879Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:31.7145754Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:31.7154202Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:31.7161597Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:31.7208701Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-11-23T01:34:33.7597994Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-11-23T01:34:34.8646500Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-11-23T01:34:37.3047467Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-11-23T01:34:39.2648235Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-11-23T01:34:39.2928637Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-11-23T01:34:39.3928847Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-11-23T01:34:39.4363748Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-11-23T01:34:39.4382228Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:39.4437063Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-11-23T01:34:40.3542544Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:34:40.5469028Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-11-23T01:34:40.5542606Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-11-23T01:34:40.6062113Z Entering 'android/libs/fbjni' 2022-11-23T01:34:40.6150892Z Entering 'third_party/FP16' 2022-11-23T01:34:40.6235830Z Entering 'third_party/FXdiv' 2022-11-23T01:34:40.6308568Z Entering 'third_party/NNPACK' 2022-11-23T01:34:40.6386446Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:40.6466690Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:40.6547896Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:40.6650330Z Entering 'third_party/benchmark' 2022-11-23T01:34:40.6734282Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:40.6823060Z Entering 'third_party/cub' 2022-11-23T01:34:40.6907801Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:40.7008093Z Entering 'third_party/cutlass' 2022-11-23T01:34:40.7107967Z Entering 'third_party/eigen' 2022-11-23T01:34:40.7195703Z Entering 'third_party/fbgemm' 2022-11-23T01:34:40.7281204Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:40.7364549Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:40.7453321Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:40.7539790Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:40.7629232Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:40.7723768Z Entering 'third_party/fmt' 2022-11-23T01:34:40.7810939Z Entering 'third_party/foxi' 2022-11-23T01:34:40.7891389Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:40.7977724Z Entering 'third_party/gloo' 2022-11-23T01:34:40.8067290Z Entering 'third_party/googletest' 2022-11-23T01:34:40.8155454Z Entering 'third_party/ideep' 2022-11-23T01:34:40.8241604Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:40.8335798Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:40.8446330Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:40.8546491Z Entering 'third_party/ittapi' 2022-11-23T01:34:40.8637704Z Entering 'third_party/kineto' 2022-11-23T01:34:40.8748636Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:40.8852186Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:40.8946590Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:40.9063957Z Entering 'third_party/neon2sse' 2022-11-23T01:34:40.9156414Z Entering 'third_party/nlohmann' 2022-11-23T01:34:40.9249333Z Entering 'third_party/onnx' 2022-11-23T01:34:40.9366941Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:40.9471131Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:40.9565938Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:40.9695166Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:40.9838174Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:40.9929636Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:41.0019008Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:41.0124880Z Entering 'third_party/pocketfft' 2022-11-23T01:34:41.0215310Z Entering 'third_party/protobuf' 2022-11-23T01:34:41.0311166Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:41.0399639Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:41.0491945Z Entering 'third_party/psimd' 2022-11-23T01:34:41.0577300Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:41.0664614Z Entering 'third_party/pybind11' 2022-11-23T01:34:41.0754984Z Entering 'third_party/python-enum' 2022-11-23T01:34:41.0842932Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:41.0930008Z Entering 'third_party/python-six' 2022-11-23T01:34:41.1012332Z Entering 'third_party/sleef' 2022-11-23T01:34:41.1097701Z Entering 'third_party/tbb' 2022-11-23T01:34:41.1192976Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:41.1273872Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:41.1356623Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:41.1439875Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:41.1524278Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:41.1609686Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:41.1708522Z Entering 'third_party/zstd' 2022-11-23T01:34:41.1822730Z ##[endgroup] 2022-11-23T01:34:41.1826947Z ##[group]Persisting credentials for submodules 2022-11-23T01:34:41.1839752Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-11-23T01:34:41.2372645Z Entering 'android/libs/fbjni' 2022-11-23T01:34:41.2457428Z Entering 'third_party/FP16' 2022-11-23T01:34:41.2539897Z Entering 'third_party/FXdiv' 2022-11-23T01:34:41.2620255Z Entering 'third_party/NNPACK' 2022-11-23T01:34:41.2689075Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:41.2759107Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:41.2835796Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:41.2927540Z Entering 'third_party/benchmark' 2022-11-23T01:34:41.3002141Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:41.3085702Z Entering 'third_party/cub' 2022-11-23T01:34:41.3168863Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:41.3262158Z Entering 'third_party/cutlass' 2022-11-23T01:34:41.3358012Z Entering 'third_party/eigen' 2022-11-23T01:34:41.3438154Z Entering 'third_party/fbgemm' 2022-11-23T01:34:41.3513214Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:41.3597068Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:41.3682117Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:41.3759413Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:41.3847157Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:41.3939009Z Entering 'third_party/fmt' 2022-11-23T01:34:41.4024940Z Entering 'third_party/foxi' 2022-11-23T01:34:41.4111291Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:41.4198848Z Entering 'third_party/gloo' 2022-11-23T01:34:41.4285001Z Entering 'third_party/googletest' 2022-11-23T01:34:41.4367578Z Entering 'third_party/ideep' 2022-11-23T01:34:41.4451919Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:41.4543169Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:41.4647022Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:41.4733924Z Entering 'third_party/ittapi' 2022-11-23T01:34:41.4817810Z Entering 'third_party/kineto' 2022-11-23T01:34:41.4902437Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:41.4986143Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:41.5076749Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:41.5162679Z Entering 'third_party/neon2sse' 2022-11-23T01:34:41.5248620Z Entering 'third_party/nlohmann' 2022-11-23T01:34:41.5336792Z Entering 'third_party/onnx' 2022-11-23T01:34:41.5457313Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:41.5540950Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:41.5626248Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:41.5700962Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:41.5798593Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:41.5884431Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:41.5969623Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:41.6074106Z Entering 'third_party/pocketfft' 2022-11-23T01:34:41.6159985Z Entering 'third_party/protobuf' 2022-11-23T01:34:41.6256519Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:41.6341214Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:41.6429914Z Entering 'third_party/psimd' 2022-11-23T01:34:41.6508909Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:41.6594142Z Entering 'third_party/pybind11' 2022-11-23T01:34:41.6679617Z Entering 'third_party/python-enum' 2022-11-23T01:34:41.6764026Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:41.6847083Z Entering 'third_party/python-six' 2022-11-23T01:34:41.6932483Z Entering 'third_party/sleef' 2022-11-23T01:34:41.7015937Z Entering 'third_party/tbb' 2022-11-23T01:34:41.7108229Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:41.7194635Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:41.7280397Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:41.7363533Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:41.7449412Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:41.7530367Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:41.7624689Z Entering 'third_party/zstd' 2022-11-23T01:34:41.7742624Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-11-23T01:34:41.8255982Z Entering 'android/libs/fbjni' 2022-11-23T01:34:41.8331689Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-11-23T01:34:41.8371554Z Entering 'third_party/FP16' 2022-11-23T01:34:41.8446061Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-11-23T01:34:41.8486776Z Entering 'third_party/FXdiv' 2022-11-23T01:34:41.8563344Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-11-23T01:34:41.8604318Z Entering 'third_party/NNPACK' 2022-11-23T01:34:41.8681845Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-11-23T01:34:41.8723928Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:41.8800774Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-11-23T01:34:41.8843567Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:41.8920578Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2022-11-23T01:34:41.8963090Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:41.9038469Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-11-23T01:34:41.9108299Z Entering 'third_party/benchmark' 2022-11-23T01:34:41.9185514Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:34:41.9229047Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:41.9294628Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:34:41.9338756Z Entering 'third_party/cub' 2022-11-23T01:34:41.9410696Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-11-23T01:34:41.9446396Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:41.9522325Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-11-23T01:34:41.9573669Z Entering 'third_party/cutlass' 2022-11-23T01:34:41.9648825Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2022-11-23T01:34:41.9704982Z Entering 'third_party/eigen' 2022-11-23T01:34:41.9778649Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-11-23T01:34:41.9825679Z Entering 'third_party/fbgemm' 2022-11-23T01:34:41.9901628Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-11-23T01:34:41.9942837Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:42.0021075Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-11-23T01:34:42.0063498Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:42.0140357Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:34:42.0183260Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:42.0245673Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:34:42.0278969Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:42.0358656Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2022-11-23T01:34:42.0404900Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:42.0483226Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-11-23T01:34:42.0530880Z Entering 'third_party/fmt' 2022-11-23T01:34:42.0607113Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-11-23T01:34:42.0649659Z Entering 'third_party/foxi' 2022-11-23T01:34:42.0725551Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-11-23T01:34:42.0767418Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:42.0845075Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-11-23T01:34:42.0886746Z Entering 'third_party/gloo' 2022-11-23T01:34:42.0964060Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-11-23T01:34:42.1006679Z Entering 'third_party/googletest' 2022-11-23T01:34:42.1083930Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:34:42.1125206Z Entering 'third_party/ideep' 2022-11-23T01:34:42.1203331Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-11-23T01:34:42.1242506Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:42.1318445Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-11-23T01:34:42.1366103Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:42.1442204Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-11-23T01:34:42.1503702Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:42.1581458Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-11-23T01:34:42.1622810Z Entering 'third_party/ittapi' 2022-11-23T01:34:42.1699297Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2022-11-23T01:34:42.1741280Z Entering 'third_party/kineto' 2022-11-23T01:34:42.1817904Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-11-23T01:34:42.1859081Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:42.1936046Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-11-23T01:34:42.1976404Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:42.2048891Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-11-23T01:34:42.2092421Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:42.2170070Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-11-23T01:34:42.2214100Z Entering 'third_party/neon2sse' 2022-11-23T01:34:42.2289932Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-11-23T01:34:42.2331855Z Entering 'third_party/nlohmann' 2022-11-23T01:34:42.2404433Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2022-11-23T01:34:42.2448543Z Entering 'third_party/onnx' 2022-11-23T01:34:42.2513208Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:34:42.2589895Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:42.2666355Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:34:42.2707549Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:42.2777520Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:34:42.2824855Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:42.2902116Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-11-23T01:34:42.2942597Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:42.3012142Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:34:42.3061408Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:42.3139105Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:34:42.3176010Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:42.3257083Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:34:42.3299066Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:42.3380683Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:34:42.3433083Z Entering 'third_party/pocketfft' 2022-11-23T01:34:42.3510562Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-11-23T01:34:42.3550379Z Entering 'third_party/protobuf' 2022-11-23T01:34:42.3626104Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-11-23T01:34:42.3674163Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:42.3751558Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:34:42.3790226Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:42.3863069Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:34:42.3906980Z Entering 'third_party/psimd' 2022-11-23T01:34:42.3982450Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-11-23T01:34:42.4021820Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:42.4099857Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-11-23T01:34:42.4142044Z Entering 'third_party/pybind11' 2022-11-23T01:34:42.4219476Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:34:42.4262415Z Entering 'third_party/python-enum' 2022-11-23T01:34:42.4337621Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-11-23T01:34:42.4378747Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:42.4452831Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-11-23T01:34:42.4495109Z Entering 'third_party/python-six' 2022-11-23T01:34:42.4571845Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-11-23T01:34:42.4613912Z Entering 'third_party/sleef' 2022-11-23T01:34:42.4688788Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-11-23T01:34:42.4732051Z Entering 'third_party/tbb' 2022-11-23T01:34:42.4809136Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-11-23T01:34:42.4857235Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:42.4935450Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-11-23T01:34:42.4977461Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:42.5053598Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:34:42.5094269Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:42.5168852Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-11-23T01:34:42.5210060Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:42.5285610Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-11-23T01:34:42.5327073Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:42.5404018Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:34:42.5443288Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:42.5523260Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:34:42.5572300Z Entering 'third_party/zstd' 2022-11-23T01:34:42.5648979Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-11-23T01:34:42.6046437Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-11-23T01:34:42.6575514Z Entering 'android/libs/fbjni' 2022-11-23T01:34:42.6663017Z Entering 'third_party/FP16' 2022-11-23T01:34:42.6749739Z Entering 'third_party/FXdiv' 2022-11-23T01:34:42.6835117Z Entering 'third_party/NNPACK' 2022-11-23T01:34:42.6921934Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:42.7011063Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:42.7099323Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:42.7212830Z Entering 'third_party/benchmark' 2022-11-23T01:34:42.7302517Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:42.7388310Z Entering 'third_party/cub' 2022-11-23T01:34:42.7472369Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:42.7571953Z Entering 'third_party/cutlass' 2022-11-23T01:34:42.7668027Z Entering 'third_party/eigen' 2022-11-23T01:34:42.7753532Z Entering 'third_party/fbgemm' 2022-11-23T01:34:42.7833922Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:42.7920935Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:42.8009195Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:42.8084003Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:42.8169696Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:42.8264169Z Entering 'third_party/fmt' 2022-11-23T01:34:42.8347129Z Entering 'third_party/foxi' 2022-11-23T01:34:42.8430338Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:42.8515972Z Entering 'third_party/gloo' 2022-11-23T01:34:42.8605357Z Entering 'third_party/googletest' 2022-11-23T01:34:42.8692982Z Entering 'third_party/ideep' 2022-11-23T01:34:42.8777548Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:42.8872328Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:42.8981725Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:42.9071997Z Entering 'third_party/ittapi' 2022-11-23T01:34:42.9162271Z Entering 'third_party/kineto' 2022-11-23T01:34:42.9247736Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:42.9338217Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:42.9428115Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:42.9519005Z Entering 'third_party/neon2sse' 2022-11-23T01:34:42.9607411Z Entering 'third_party/nlohmann' 2022-11-23T01:34:42.9701836Z Entering 'third_party/onnx' 2022-11-23T01:34:42.9814209Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:42.9900526Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:42.9991754Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:43.0079625Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:43.0173446Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:43.0259377Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:43.0343628Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:43.0434205Z Entering 'third_party/pocketfft' 2022-11-23T01:34:43.0522378Z Entering 'third_party/protobuf' 2022-11-23T01:34:43.0618182Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:43.0705880Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:43.0795998Z Entering 'third_party/psimd' 2022-11-23T01:34:43.0878400Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:43.0963116Z Entering 'third_party/pybind11' 2022-11-23T01:34:43.1054597Z Entering 'third_party/python-enum' 2022-11-23T01:34:43.1143340Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:43.1230904Z Entering 'third_party/python-six' 2022-11-23T01:34:43.1314343Z Entering 'third_party/sleef' 2022-11-23T01:34:43.1399185Z Entering 'third_party/tbb' 2022-11-23T01:34:43.1494329Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:43.1581038Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:43.1670018Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:43.1750335Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:43.1834417Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:43.1915736Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:43.2006066Z Entering 'third_party/zstd' 2022-11-23T01:34:43.2126153Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2022-11-23T01:34:43.2659873Z Entering 'android/libs/fbjni' 2022-11-23T01:34:43.2750520Z Entering 'third_party/FP16' 2022-11-23T01:34:43.2829784Z Entering 'third_party/FXdiv' 2022-11-23T01:34:43.2907949Z Entering 'third_party/NNPACK' 2022-11-23T01:34:43.2993083Z Entering 'third_party/QNNPACK' 2022-11-23T01:34:43.3082710Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:34:43.3165804Z Entering 'third_party/XNNPACK' 2022-11-23T01:34:43.3276501Z Entering 'third_party/benchmark' 2022-11-23T01:34:43.3362882Z Entering 'third_party/cpuinfo' 2022-11-23T01:34:43.3438758Z Entering 'third_party/cub' 2022-11-23T01:34:43.3530120Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:34:43.3628059Z Entering 'third_party/cutlass' 2022-11-23T01:34:43.3733020Z Entering 'third_party/eigen' 2022-11-23T01:34:43.3823175Z Entering 'third_party/fbgemm' 2022-11-23T01:34:43.3907763Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:34:43.3990846Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:34:43.4074453Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:34:43.4156975Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:34:43.4249451Z Entering 'third_party/flatbuffers' 2022-11-23T01:34:43.4344865Z Entering 'third_party/fmt' 2022-11-23T01:34:43.4427580Z Entering 'third_party/foxi' 2022-11-23T01:34:43.4508768Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:34:43.4597022Z Entering 'third_party/gloo' 2022-11-23T01:34:43.4687751Z Entering 'third_party/googletest' 2022-11-23T01:34:43.4773861Z Entering 'third_party/ideep' 2022-11-23T01:34:43.4856715Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:34:43.4950182Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:34:43.5059509Z Entering 'third_party/ios-cmake' 2022-11-23T01:34:43.5149642Z Entering 'third_party/ittapi' 2022-11-23T01:34:43.5239725Z Entering 'third_party/kineto' 2022-11-23T01:34:43.5329384Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:34:43.5415657Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:34:43.5507758Z Entering 'third_party/nccl/nccl' 2022-11-23T01:34:43.5598478Z Entering 'third_party/neon2sse' 2022-11-23T01:34:43.5685288Z Entering 'third_party/nlohmann' 2022-11-23T01:34:43.5778290Z Entering 'third_party/onnx' 2022-11-23T01:34:43.5894460Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:34:43.5981534Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:34:43.6074405Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:34:43.6159927Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:34:43.6237635Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:34:43.6329027Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:34:43.6413869Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:34:43.6519584Z Entering 'third_party/pocketfft' 2022-11-23T01:34:43.6609717Z Entering 'third_party/protobuf' 2022-11-23T01:34:43.6707850Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:34:43.6796513Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:34:43.6884463Z Entering 'third_party/psimd' 2022-11-23T01:34:43.6974032Z Entering 'third_party/pthreadpool' 2022-11-23T01:34:43.7063212Z Entering 'third_party/pybind11' 2022-11-23T01:34:43.7154660Z Entering 'third_party/python-enum' 2022-11-23T01:34:43.7243929Z Entering 'third_party/python-peachpy' 2022-11-23T01:34:43.7332773Z Entering 'third_party/python-six' 2022-11-23T01:34:43.7419174Z Entering 'third_party/sleef' 2022-11-23T01:34:43.7508907Z Entering 'third_party/tbb' 2022-11-23T01:34:43.7602449Z Entering 'third_party/tensorpipe' 2022-11-23T01:34:43.7688171Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:34:43.7776724Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:34:43.7864722Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:34:43.7950497Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:34:43.8037146Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:34:43.8136004Z Entering 'third_party/zstd' 2022-11-23T01:34:43.8245602Z ##[endgroup] 2022-11-23T01:34:43.8344119Z [command]/usr/bin/git log -1 --format='%H' 2022-11-23T01:34:43.8413985Z '1cfd3858ac54fe3883534309081631a0a892ba3f' 2022-11-23T01:34:43.8745974Z Prepare all required actions 2022-11-23T01:34:43.8787156Z ##[group]Run ./.github/actions/setup-rocm 2022-11-23T01:34:43.8787428Z env: 2022-11-23T01:34:43.8787688Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:43.8787967Z ##[endgroup] 2022-11-23T01:34:43.8809530Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-11-23T01:34:43.8810021Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-11-23T01:34:43.8841742Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:43.8842054Z env: 2022-11-23T01:34:43.8842317Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:43.8842596Z ##[endgroup] 2022-11-23T01:34:43.8977688Z ##[group]Run cat /etc/os-release || true 2022-11-23T01:34:43.8978529Z cat /etc/os-release || true 2022-11-23T01:34:43.8979389Z cat /etc/apt/sources.list.d/rocm.list || true 2022-11-23T01:34:43.8980255Z cat /opt/rocm/.info/version || true 2022-11-23T01:34:43.8981024Z whoami 2022-11-23T01:34:43.9039078Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:43.9039804Z env: 2022-11-23T01:34:43.9040619Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:43.9041442Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:43.9042205Z ##[endgroup] 2022-11-23T01:34:43.9135824Z NAME="Ubuntu" 2022-11-23T01:34:43.9136590Z VERSION="18.04.6 LTS (Bionic Beaver)" 2022-11-23T01:34:43.9137244Z ID=ubuntu 2022-11-23T01:34:43.9137931Z ID_LIKE=debian 2022-11-23T01:34:43.9139215Z PRETTY_NAME="Ubuntu 18.04.6 LTS" 2022-11-23T01:34:43.9139664Z VERSION_ID="18.04" 2022-11-23T01:34:43.9140168Z HOME_URL="https://www.ubuntu.com/" 2022-11-23T01:34:43.9140703Z SUPPORT_URL="https://help.ubuntu.com/" 2022-11-23T01:34:43.9141325Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-11-23T01:34:43.9146528Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-11-23T01:34:43.9147613Z VERSION_CODENAME=bionic 2022-11-23T01:34:43.9148323Z UBUNTU_CODENAME=bionic 2022-11-23T01:34:43.9153479Z deb [arch=amd64] http://repo.radeon.com/rocm/apt/4.2 xenial main 2022-11-23T01:34:43.9172050Z 4.2.0-21 2022-11-23T01:34:43.9196031Z pytorchci 2022-11-23T01:34:43.9225971Z ##[group]Run rocm-smi 2022-11-23T01:34:43.9226379Z rocm-smi 2022-11-23T01:34:43.9261209Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:43.9261675Z env: 2022-11-23T01:34:43.9262054Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:43.9262558Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:43.9262975Z ##[endgroup] 2022-11-23T01:34:44.0444067Z 2022-11-23T01:34:44.0444078Z 2022-11-23T01:34:44.0444922Z ======================= ROCm System Management Interface ======================= 2022-11-23T01:34:44.0445394Z ================================= Concise Info ================================= 2022-11-23T01:34:44.0445876Z GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2022-11-23T01:34:44.0446345Z 0 37.0c 15.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:34:44.0446755Z 1 42.0c 20.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:34:44.0447140Z 2 38.0c 20.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:34:44.0447525Z 3 39.0c 16.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:34:44.0447898Z ================================================================================ 2022-11-23T01:34:44.0451933Z ============================= End of ROCm SMI Log ============================== 2022-11-23T01:34:44.0576511Z ##[group]Run rocminfo 2022-11-23T01:34:44.0577176Z rocminfo 2022-11-23T01:34:44.0639933Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:44.0641223Z env: 2022-11-23T01:34:44.0641933Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:44.0642796Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:44.0643493Z ##[endgroup] 2022-11-23T01:34:44.1729519Z ROCk module is loaded 2022-11-23T01:34:44.1730949Z ===================== 2022-11-23T01:34:44.1732150Z HSA System Attributes 2022-11-23T01:34:44.1732796Z ===================== 2022-11-23T01:34:44.1733436Z Runtime Version: 1.1 2022-11-23T01:34:44.1734179Z System Timestamp Freq.: 1000.000000MHz 2022-11-23T01:34:44.1735393Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2022-11-23T01:34:44.1737170Z Machine Model: LARGE 2022-11-23T01:34:44.1738551Z System Endianness: LITTLE 2022-11-23T01:34:44.1739331Z 2022-11-23T01:34:44.1739683Z ========== 2022-11-23T01:34:44.1740673Z HSA Agents 2022-11-23T01:34:44.1741645Z ========== 2022-11-23T01:34:44.1742507Z ******* 2022-11-23T01:34:44.1743448Z Agent 1 2022-11-23T01:34:44.1744351Z ******* 2022-11-23T01:34:44.1745502Z Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:34:44.1747261Z Uuid: CPU-XX 2022-11-23T01:34:44.1748664Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:34:44.1750304Z Vendor Name: CPU 2022-11-23T01:34:44.1751919Z Feature: None specified 2022-11-23T01:34:44.1752793Z Profile: FULL_PROFILE 2022-11-23T01:34:44.1753898Z Float Round Mode: NEAR 2022-11-23T01:34:44.1754722Z Max Queue Number: 0(0x0) 2022-11-23T01:34:44.1755539Z Queue Min Size: 0(0x0) 2022-11-23T01:34:44.1756329Z Queue Max Size: 0(0x0) 2022-11-23T01:34:44.1757124Z Queue Type: MULTI 2022-11-23T01:34:44.1757900Z Node: 0 2022-11-23T01:34:44.1758643Z Device Type: CPU 2022-11-23T01:34:44.1759348Z Cache Info: 2022-11-23T01:34:44.1760080Z L1: 32768(0x8000) KB 2022-11-23T01:34:44.1761095Z Chip ID: 0(0x0) 2022-11-23T01:34:44.1761886Z Cacheline Size: 64(0x40) 2022-11-23T01:34:44.1762699Z Max Clock Freq. (MHz): 3700 2022-11-23T01:34:44.1763480Z BDFID: 0 2022-11-23T01:34:44.1764237Z Internal Node ID: 0 2022-11-23T01:34:44.1765013Z Compute Unit: 28 2022-11-23T01:34:44.1765781Z SIMDs per CU: 0 2022-11-23T01:34:44.1766572Z Shader Engines: 0 2022-11-23T01:34:44.1767389Z Shader Arrs. per Eng.: 0 2022-11-23T01:34:44.1768219Z WatchPts on Addr. Ranges:1 2022-11-23T01:34:44.1768936Z Features: None 2022-11-23T01:34:44.1769595Z Pool Info: 2022-11-23T01:34:44.1770259Z Pool 1 2022-11-23T01:34:44.1771072Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:34:44.1771969Z Size: 263817324(0xfb9886c) KB 2022-11-23T01:34:44.1772827Z Allocatable: TRUE 2022-11-23T01:34:44.1773653Z Alloc Granule: 4KB 2022-11-23T01:34:44.1774485Z Alloc Alignment: 4KB 2022-11-23T01:34:44.1775344Z Accessible by all: TRUE 2022-11-23T01:34:44.1776076Z Pool 2 2022-11-23T01:34:44.1777025Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:44.1777892Z Size: 263817324(0xfb9886c) KB 2022-11-23T01:34:44.1778685Z Allocatable: TRUE 2022-11-23T01:34:44.1779514Z Alloc Granule: 4KB 2022-11-23T01:34:44.1780348Z Alloc Alignment: 4KB 2022-11-23T01:34:44.1781183Z Accessible by all: TRUE 2022-11-23T01:34:44.1781926Z ISA Info: 2022-11-23T01:34:44.1782568Z N/A 2022-11-23T01:34:44.1783157Z ******* 2022-11-23T01:34:44.1783762Z Agent 2 2022-11-23T01:34:44.1784360Z ******* 2022-11-23T01:34:44.1785119Z Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:34:44.1786257Z Uuid: CPU-XX 2022-11-23T01:34:44.1787158Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:34:44.1788021Z Vendor Name: CPU 2022-11-23T01:34:44.1788856Z Feature: None specified 2022-11-23T01:34:44.1789702Z Profile: FULL_PROFILE 2022-11-23T01:34:44.1790530Z Float Round Mode: NEAR 2022-11-23T01:34:44.1791341Z Max Queue Number: 0(0x0) 2022-11-23T01:34:44.1792273Z Queue Min Size: 0(0x0) 2022-11-23T01:34:44.1793030Z Queue Max Size: 0(0x0) 2022-11-23T01:34:44.1793820Z Queue Type: MULTI 2022-11-23T01:34:44.1794574Z Node: 1 2022-11-23T01:34:44.1795325Z Device Type: CPU 2022-11-23T01:34:44.1796034Z Cache Info: 2022-11-23T01:34:44.1796774Z L1: 32768(0x8000) KB 2022-11-23T01:34:44.1797566Z Chip ID: 0(0x0) 2022-11-23T01:34:44.1798325Z Cacheline Size: 64(0x40) 2022-11-23T01:34:44.1799117Z Max Clock Freq. (MHz): 3700 2022-11-23T01:34:44.1799884Z BDFID: 0 2022-11-23T01:34:44.1800786Z Internal Node ID: 1 2022-11-23T01:34:44.1801571Z Compute Unit: 28 2022-11-23T01:34:44.1802346Z SIMDs per CU: 0 2022-11-23T01:34:44.1803101Z Shader Engines: 0 2022-11-23T01:34:44.1803897Z Shader Arrs. per Eng.: 0 2022-11-23T01:34:44.1804736Z WatchPts on Addr. Ranges:1 2022-11-23T01:34:44.1805486Z Features: None 2022-11-23T01:34:44.1806149Z Pool Info: 2022-11-23T01:34:44.1806808Z Pool 1 2022-11-23T01:34:44.1807605Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:34:44.1808490Z Size: 264203864(0xfbf6e58) KB 2022-11-23T01:34:44.1809332Z Allocatable: TRUE 2022-11-23T01:34:44.1810172Z Alloc Granule: 4KB 2022-11-23T01:34:44.1811002Z Alloc Alignment: 4KB 2022-11-23T01:34:44.1811837Z Accessible by all: TRUE 2022-11-23T01:34:44.1812539Z Pool 2 2022-11-23T01:34:44.1813327Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:44.1814202Z Size: 264203864(0xfbf6e58) KB 2022-11-23T01:34:44.1815155Z Allocatable: TRUE 2022-11-23T01:34:44.1815992Z Alloc Granule: 4KB 2022-11-23T01:34:44.1816813Z Alloc Alignment: 4KB 2022-11-23T01:34:44.1817628Z Accessible by all: TRUE 2022-11-23T01:34:44.1818349Z ISA Info: 2022-11-23T01:34:44.1818986Z N/A 2022-11-23T01:34:44.1819613Z ******* 2022-11-23T01:34:44.1820223Z Agent 3 2022-11-23T01:34:44.1820829Z ******* 2022-11-23T01:34:44.1821507Z Name: gfx906 2022-11-23T01:34:44.1822597Z Uuid: GPU-d422688172dc76e6 2022-11-23T01:34:44.1823424Z Marketing Name: Vega 20 2022-11-23T01:34:44.1824243Z Vendor Name: AMD 2022-11-23T01:34:44.1825059Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:44.1825910Z Profile: BASE_PROFILE 2022-11-23T01:34:44.1826711Z Float Round Mode: NEAR 2022-11-23T01:34:44.1827522Z Max Queue Number: 128(0x80) 2022-11-23T01:34:44.1828314Z Queue Min Size: 4096(0x1000) 2022-11-23T01:34:44.1829105Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:44.1830033Z Queue Type: MULTI 2022-11-23T01:34:44.1830789Z Node: 2 2022-11-23T01:34:44.1831548Z Device Type: GPU 2022-11-23T01:34:44.1832223Z Cache Info: 2022-11-23T01:34:44.1832950Z L1: 16(0x10) KB 2022-11-23T01:34:44.1833741Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:44.1834538Z Cacheline Size: 64(0x40) 2022-11-23T01:34:44.1835339Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:44.1836110Z BDFID: 16128 2022-11-23T01:34:44.1836858Z Internal Node ID: 2 2022-11-23T01:34:44.1837625Z Compute Unit: 60 2022-11-23T01:34:44.1838404Z SIMDs per CU: 4 2022-11-23T01:34:44.1839224Z Shader Engines: 4 2022-11-23T01:34:44.1840018Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:44.1840939Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:44.1841748Z Features: KERNEL_DISPATCH 2022-11-23T01:34:44.1842591Z Fast F16 Operation: FALSE 2022-11-23T01:34:44.1843411Z Wavefront Size: 64(0x40) 2022-11-23T01:34:44.1844207Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:44.1844999Z Workgroup Max Size per Dimension: 2022-11-23T01:34:44.1845788Z x 1024(0x400) 2022-11-23T01:34:44.1846540Z y 1024(0x400) 2022-11-23T01:34:44.1847292Z z 1024(0x400) 2022-11-23T01:34:44.1848098Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:44.1849206Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:44.1850035Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:44.1850803Z Grid Max Size per Dimension: 2022-11-23T01:34:44.1851575Z x 4294967295(0xffffffff) 2022-11-23T01:34:44.1852484Z y 4294967295(0xffffffff) 2022-11-23T01:34:44.1853277Z z 4294967295(0xffffffff) 2022-11-23T01:34:44.1854279Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:44.1854990Z Pool Info: 2022-11-23T01:34:44.1855650Z Pool 1 2022-11-23T01:34:44.1856459Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:44.1857331Z Size: 16760832(0xffc000) KB 2022-11-23T01:34:44.1858187Z Allocatable: TRUE 2022-11-23T01:34:44.1859021Z Alloc Granule: 4KB 2022-11-23T01:34:44.1859828Z Alloc Alignment: 4KB 2022-11-23T01:34:44.1860679Z Accessible by all: FALSE 2022-11-23T01:34:44.1861421Z Pool 2 2022-11-23T01:34:44.1862171Z Segment: GROUP 2022-11-23T01:34:44.1862974Z Size: 64(0x40) KB 2022-11-23T01:34:44.1863780Z Allocatable: FALSE 2022-11-23T01:34:44.1864577Z Alloc Granule: 0KB 2022-11-23T01:34:44.1865392Z Alloc Alignment: 0KB 2022-11-23T01:34:44.1866240Z Accessible by all: FALSE 2022-11-23T01:34:44.1867111Z ISA Info: 2022-11-23T01:34:44.1867756Z ISA 1 2022-11-23T01:34:44.1868924Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:34:44.1869909Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:44.1870825Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:44.1871716Z Default Rounding Mode: NEAR 2022-11-23T01:34:44.1872587Z Default Rounding Mode: NEAR 2022-11-23T01:34:44.1873413Z Fast f16: TRUE 2022-11-23T01:34:44.1874237Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:44.1875034Z Workgroup Max Size per Dimension: 2022-11-23T01:34:44.1875812Z x 1024(0x400) 2022-11-23T01:34:44.1876597Z y 1024(0x400) 2022-11-23T01:34:44.1877353Z z 1024(0x400) 2022-11-23T01:34:44.1878164Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:44.1878944Z Grid Max Size per Dimension: 2022-11-23T01:34:44.1879726Z x 4294967295(0xffffffff) 2022-11-23T01:34:44.1880645Z y 4294967295(0xffffffff) 2022-11-23T01:34:44.1881459Z z 4294967295(0xffffffff) 2022-11-23T01:34:44.1882307Z FBarrier Max Size: 32 2022-11-23T01:34:44.1882601Z ******* 2022-11-23T01:34:44.1882859Z Agent 4 2022-11-23T01:34:44.1883115Z ******* 2022-11-23T01:34:44.1883391Z Name: gfx906 2022-11-23T01:34:44.1883833Z Uuid: GPU-17e038e172fd5d44 2022-11-23T01:34:44.1884177Z Marketing Name: Vega 20 2022-11-23T01:34:44.1884507Z Vendor Name: AMD 2022-11-23T01:34:44.1884850Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:44.1885193Z Profile: BASE_PROFILE 2022-11-23T01:34:44.1885512Z Float Round Mode: NEAR 2022-11-23T01:34:44.1885942Z Max Queue Number: 128(0x80) 2022-11-23T01:34:44.1886268Z Queue Min Size: 4096(0x1000) 2022-11-23T01:34:44.1886594Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:44.1886921Z Queue Type: MULTI 2022-11-23T01:34:44.1887238Z Node: 3 2022-11-23T01:34:44.1887555Z Device Type: GPU 2022-11-23T01:34:44.1887833Z Cache Info: 2022-11-23T01:34:44.1888136Z L1: 16(0x10) KB 2022-11-23T01:34:44.1888461Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:44.1888786Z Cacheline Size: 64(0x40) 2022-11-23T01:34:44.1889115Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:44.1889433Z BDFID: 17152 2022-11-23T01:34:44.1889730Z Internal Node ID: 3 2022-11-23T01:34:44.1890047Z Compute Unit: 60 2022-11-23T01:34:44.1890361Z SIMDs per CU: 4 2022-11-23T01:34:44.1890680Z Shader Engines: 4 2022-11-23T01:34:44.1891007Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:44.1891400Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:44.1891708Z Features: KERNEL_DISPATCH 2022-11-23T01:34:44.1892044Z Fast F16 Operation: FALSE 2022-11-23T01:34:44.1892379Z Wavefront Size: 64(0x40) 2022-11-23T01:34:44.1892712Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:44.1893033Z Workgroup Max Size per Dimension: 2022-11-23T01:34:44.1893360Z x 1024(0x400) 2022-11-23T01:34:44.1893652Z y 1024(0x400) 2022-11-23T01:34:44.1893960Z z 1024(0x400) 2022-11-23T01:34:44.1894277Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:44.1894711Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:44.1895057Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:44.1895379Z Grid Max Size per Dimension: 2022-11-23T01:34:44.1895677Z x 4294967295(0xffffffff) 2022-11-23T01:34:44.1896001Z y 4294967295(0xffffffff) 2022-11-23T01:34:44.1896317Z z 4294967295(0xffffffff) 2022-11-23T01:34:44.1896653Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:44.1896955Z Pool Info: 2022-11-23T01:34:44.1897227Z Pool 1 2022-11-23T01:34:44.1897540Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:44.1897894Z Size: 16760832(0xffc000) KB 2022-11-23T01:34:44.1898234Z Allocatable: TRUE 2022-11-23T01:34:44.1898576Z Alloc Granule: 4KB 2022-11-23T01:34:44.1898960Z Alloc Alignment: 4KB 2022-11-23T01:34:44.1899351Z Accessible by all: FALSE 2022-11-23T01:34:44.1899666Z Pool 2 2022-11-23T01:34:44.1899980Z Segment: GROUP 2022-11-23T01:34:44.1900291Z Size: 64(0x40) KB 2022-11-23T01:34:44.1900628Z Allocatable: FALSE 2022-11-23T01:34:44.1901032Z Alloc Granule: 0KB 2022-11-23T01:34:44.1901371Z Alloc Alignment: 0KB 2022-11-23T01:34:44.1901715Z Accessible by all: FALSE 2022-11-23T01:34:44.1902019Z ISA Info: 2022-11-23T01:34:44.1902288Z ISA 1 2022-11-23T01:34:44.1902727Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:34:44.1903137Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:44.1903516Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:44.1903877Z Default Rounding Mode: NEAR 2022-11-23T01:34:44.1904234Z Default Rounding Mode: NEAR 2022-11-23T01:34:44.1904574Z Fast f16: TRUE 2022-11-23T01:34:44.1904890Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:44.1905224Z Workgroup Max Size per Dimension: 2022-11-23T01:34:44.1905553Z x 1024(0x400) 2022-11-23T01:34:44.1905867Z y 1024(0x400) 2022-11-23T01:34:44.1906183Z z 1024(0x400) 2022-11-23T01:34:44.1906516Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:44.1906820Z Grid Max Size per Dimension: 2022-11-23T01:34:44.1907192Z x 4294967295(0xffffffff) 2022-11-23T01:34:44.1907524Z y 4294967295(0xffffffff) 2022-11-23T01:34:44.1907854Z z 4294967295(0xffffffff) 2022-11-23T01:34:44.1908189Z FBarrier Max Size: 32 2022-11-23T01:34:44.1908480Z ******* 2022-11-23T01:34:44.1908721Z Agent 5 2022-11-23T01:34:44.1908972Z ******* 2022-11-23T01:34:44.1909263Z Name: gfx906 2022-11-23T01:34:44.1909697Z Uuid: GPU-6220594172da5ee8 2022-11-23T01:34:44.1910035Z Marketing Name: Vega 20 2022-11-23T01:34:44.1910365Z Vendor Name: AMD 2022-11-23T01:34:44.1910684Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:44.1911037Z Profile: BASE_PROFILE 2022-11-23T01:34:44.1911372Z Float Round Mode: NEAR 2022-11-23T01:34:44.1911701Z Max Queue Number: 128(0x80) 2022-11-23T01:34:44.1912030Z Queue Min Size: 4096(0x1000) 2022-11-23T01:34:44.1912352Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:44.1912677Z Queue Type: MULTI 2022-11-23T01:34:44.1912972Z Node: 4 2022-11-23T01:34:44.1913342Z Device Type: GPU 2022-11-23T01:34:44.1913634Z Cache Info: 2022-11-23T01:34:44.1913938Z L1: 16(0x10) KB 2022-11-23T01:34:44.1914267Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:44.1914685Z Cacheline Size: 64(0x40) 2022-11-23T01:34:44.1915148Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:44.1915545Z BDFID: 17920 2022-11-23T01:34:44.1915956Z Internal Node ID: 4 2022-11-23T01:34:44.1916296Z Compute Unit: 60 2022-11-23T01:34:44.1916747Z SIMDs per CU: 4 2022-11-23T01:34:44.1951087Z Shader Engines: 4 2022-11-23T01:34:44.1951504Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:44.1951882Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:44.1952235Z Features: KERNEL_DISPATCH 2022-11-23T01:34:44.1952602Z Fast F16 Operation: FALSE 2022-11-23T01:34:44.1952966Z Wavefront Size: 64(0x40) 2022-11-23T01:34:44.1953335Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:44.1953681Z Workgroup Max Size per Dimension: 2022-11-23T01:34:44.1953999Z x 1024(0x400) 2022-11-23T01:34:44.1954330Z y 1024(0x400) 2022-11-23T01:34:44.1954658Z z 1024(0x400) 2022-11-23T01:34:44.1955011Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:44.1955557Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:44.1955928Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:44.1956242Z Grid Max Size per Dimension: 2022-11-23T01:34:44.1956568Z x 4294967295(0xffffffff) 2022-11-23T01:34:44.1956907Z y 4294967295(0xffffffff) 2022-11-23T01:34:44.1957479Z z 4294967295(0xffffffff) 2022-11-23T01:34:44.1957833Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:44.1958152Z Pool Info: 2022-11-23T01:34:44.1958419Z Pool 1 2022-11-23T01:34:44.1958765Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:44.1959130Z Size: 16760832(0xffc000) KB 2022-11-23T01:34:44.1959491Z Allocatable: TRUE 2022-11-23T01:34:44.1959846Z Alloc Granule: 4KB 2022-11-23T01:34:44.1960283Z Alloc Alignment: 4KB 2022-11-23T01:34:44.1960619Z Accessible by all: FALSE 2022-11-23T01:34:44.1960935Z Pool 2 2022-11-23T01:34:44.1961260Z Segment: GROUP 2022-11-23T01:34:44.1961615Z Size: 64(0x40) KB 2022-11-23T01:34:44.1962038Z Allocatable: FALSE 2022-11-23T01:34:44.1962392Z Alloc Granule: 0KB 2022-11-23T01:34:44.1962721Z Alloc Alignment: 0KB 2022-11-23T01:34:44.1963080Z Accessible by all: FALSE 2022-11-23T01:34:44.1963397Z ISA Info: 2022-11-23T01:34:44.1963677Z ISA 1 2022-11-23T01:34:44.1964170Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:34:44.1964589Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:44.1964972Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:44.1965323Z Default Rounding Mode: NEAR 2022-11-23T01:34:44.1965688Z Default Rounding Mode: NEAR 2022-11-23T01:34:44.1966035Z Fast f16: TRUE 2022-11-23T01:34:44.1966369Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:44.1966714Z Workgroup Max Size per Dimension: 2022-11-23T01:34:44.1967056Z x 1024(0x400) 2022-11-23T01:34:44.1967355Z y 1024(0x400) 2022-11-23T01:34:44.1967776Z z 1024(0x400) 2022-11-23T01:34:44.1968138Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:44.1968466Z Grid Max Size per Dimension: 2022-11-23T01:34:44.1968801Z x 4294967295(0xffffffff) 2022-11-23T01:34:44.1969134Z y 4294967295(0xffffffff) 2022-11-23T01:34:44.1969450Z z 4294967295(0xffffffff) 2022-11-23T01:34:44.1969797Z FBarrier Max Size: 32 2022-11-23T01:34:44.1970103Z ******* 2022-11-23T01:34:44.1970360Z Agent 6 2022-11-23T01:34:44.1970637Z ******* 2022-11-23T01:34:44.1970926Z Name: gfx906 2022-11-23T01:34:44.1971361Z Uuid: GPU-49d058c172fd5d11 2022-11-23T01:34:44.1971711Z Marketing Name: Vega 20 2022-11-23T01:34:44.1972055Z Vendor Name: AMD 2022-11-23T01:34:44.1972412Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:44.1972771Z Profile: BASE_PROFILE 2022-11-23T01:34:44.1973115Z Float Round Mode: NEAR 2022-11-23T01:34:44.1973441Z Max Queue Number: 128(0x80) 2022-11-23T01:34:44.1973845Z Queue Min Size: 4096(0x1000) 2022-11-23T01:34:44.1974180Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:44.1974524Z Queue Type: MULTI 2022-11-23T01:34:44.1974849Z Node: 5 2022-11-23T01:34:44.1975180Z Device Type: GPU 2022-11-23T01:34:44.1975485Z Cache Info: 2022-11-23T01:34:44.1975782Z L1: 16(0x10) KB 2022-11-23T01:34:44.1976114Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:44.1976450Z Cacheline Size: 64(0x40) 2022-11-23T01:34:44.1976790Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:44.1977164Z BDFID: 18688 2022-11-23T01:34:44.1999080Z Internal Node ID: 5 2022-11-23T01:34:44.1999447Z Compute Unit: 60 2022-11-23T01:34:44.1999791Z SIMDs per CU: 4 2022-11-23T01:34:44.2000241Z Shader Engines: 4 2022-11-23T01:34:44.2000596Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:44.2000963Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:44.2001320Z Features: KERNEL_DISPATCH 2022-11-23T01:34:44.2001656Z Fast F16 Operation: FALSE 2022-11-23T01:34:44.2002017Z Wavefront Size: 64(0x40) 2022-11-23T01:34:44.2002370Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:44.2002719Z Workgroup Max Size per Dimension: 2022-11-23T01:34:44.2003065Z x 1024(0x400) 2022-11-23T01:34:44.2003396Z y 1024(0x400) 2022-11-23T01:34:44.2003702Z z 1024(0x400) 2022-11-23T01:34:44.2004042Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:44.2004580Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:44.2004948Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:44.2005467Z Grid Max Size per Dimension: 2022-11-23T01:34:44.2005807Z x 4294967295(0xffffffff) 2022-11-23T01:34:44.2006148Z y 4294967295(0xffffffff) 2022-11-23T01:34:44.2006469Z z 4294967295(0xffffffff) 2022-11-23T01:34:44.2006816Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:44.2007133Z Pool Info: 2022-11-23T01:34:44.2007426Z Pool 1 2022-11-23T01:34:44.2007787Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:44.2008209Z Size: 16760832(0xffc000) KB 2022-11-23T01:34:44.2008538Z Allocatable: TRUE 2022-11-23T01:34:44.2008916Z Alloc Granule: 4KB 2022-11-23T01:34:44.2009272Z Alloc Alignment: 4KB 2022-11-23T01:34:44.2009634Z Accessible by all: FALSE 2022-11-23T01:34:44.2009952Z Pool 2 2022-11-23T01:34:44.2010314Z Segment: GROUP 2022-11-23T01:34:44.2010646Z Size: 64(0x40) KB 2022-11-23T01:34:44.2011019Z Allocatable: FALSE 2022-11-23T01:34:44.2011435Z Alloc Granule: 0KB 2022-11-23T01:34:44.2011893Z Alloc Alignment: 0KB 2022-11-23T01:34:44.2031899Z Accessible by all: FALSE 2022-11-23T01:34:44.2032757Z ISA Info: 2022-11-23T01:34:44.2033023Z ISA 1 2022-11-23T01:34:44.2033593Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:34:44.2034015Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:44.2034405Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:44.2034772Z Default Rounding Mode: NEAR 2022-11-23T01:34:44.2035119Z Default Rounding Mode: NEAR 2022-11-23T01:34:44.2035458Z Fast f16: TRUE 2022-11-23T01:34:44.2035781Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:44.2036123Z Workgroup Max Size per Dimension: 2022-11-23T01:34:44.2036463Z x 1024(0x400) 2022-11-23T01:34:44.2036788Z y 1024(0x400) 2022-11-23T01:34:44.2037108Z z 1024(0x400) 2022-11-23T01:34:44.2037446Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:44.2037756Z Grid Max Size per Dimension: 2022-11-23T01:34:44.2038086Z x 4294967295(0xffffffff) 2022-11-23T01:34:44.2038423Z y 4294967295(0xffffffff) 2022-11-23T01:34:44.2038757Z z 4294967295(0xffffffff) 2022-11-23T01:34:44.2039101Z FBarrier Max Size: 32 2022-11-23T01:34:44.2039408Z *** Done *** 2022-11-23T01:34:44.2082414Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-11-23T01:34:44.2082819Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-11-23T01:34:44.2083149Z if [[ "x$ngpu" != "x2" && "x$ngpu" != "x4" ]]; then 2022-11-23T01:34:44.2083444Z  if [[ $ngpu -eq 0 ]]; then 2022-11-23T01:34:44.2083776Z  echo "Error: Failed to detect any GPUs on the runner" 2022-11-23T01:34:44.2084073Z  else 2022-11-23T01:34:44.2084413Z  echo "Error: Detected $ngpu GPUs on the runner, when only 2 or 4 were expected" 2022-11-23T01:34:44.2084838Z  fi 2022-11-23T01:34:44.2085271Z  echo "Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2022-11-23T01:34:44.2085688Z  exit 1 2022-11-23T01:34:44.2085906Z fi 2022-11-23T01:34:44.2115145Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:44.2115430Z env: 2022-11-23T01:34:44.2115661Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:44.2115973Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:44.2116244Z ##[endgroup] 2022-11-23T01:34:44.3126868Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:34:44.3127912Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:34:44.3128851Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:34:44.3188158Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:44.3188888Z env: 2022-11-23T01:34:44.3189487Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:44.3190273Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:44.3190974Z ##[endgroup] 2022-11-23T01:34:44.3407924Z ##[group]Run # Examine the runner name. If it ends with "-2", this is the second runner on the host. 2022-11-23T01:34:44.3408634Z # Examine the runner name. If it ends with "-2", this is the second runner on the host. 2022-11-23T01:34:44.3409192Z if [[ worker-rocm-amd-106 == *-2 ]]; then 2022-11-23T01:34:44.3409833Z  # select the last two GPUs on the host 2022-11-23T01:34:44.3410586Z  echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD130 --device=/dev/dri/renderD131 --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-11-23T01:34:44.3411227Z else 2022-11-23T01:34:44.3411631Z  # select the first two GPUs on the host 2022-11-23T01:34:44.3412376Z  echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-11-23T01:34:44.3413023Z fi 2022-11-23T01:34:44.3444385Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:44.3444708Z env: 2022-11-23T01:34:44.3444967Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:44.3445284Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:44.3445587Z ##[endgroup] 2022-11-23T01:34:44.3585197Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2022-11-23T01:34:44.3586072Z with: 2022-11-23T01:34:44.3587255Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:34:44.3588419Z env: 2022-11-23T01:34:44.3589031Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:44.3589832Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:44.3591092Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:44.3592189Z ##[endgroup] 2022-11-23T01:34:44.3627796Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:34:44.3628748Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:34:44.3629710Z # ignore output since only exit code is used for conditional 2022-11-23T01:34:44.3630784Z # only pull docker image if it's not available locally 2022-11-23T01:34:44.3631863Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2022-11-23T01:34:44.3632855Z  retry docker pull "${DOCKER_IMAGE}" 2022-11-23T01:34:44.3633522Z fi 2022-11-23T01:34:44.3664201Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:44.3664519Z env: 2022-11-23T01:34:44.3664782Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:44.3665111Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:44.3665624Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:44.3666315Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:34:44.3666786Z ##[endgroup] 2022-11-23T01:34:44.4368020Z ##[group]Run python3 -m pip install psutil==5.9.1 2022-11-23T01:34:44.4368839Z python3 -m pip install psutil==5.9.1 2022-11-23T01:34:44.4369550Z python3 -m pip install pynvml==11.4.1 2022-11-23T01:34:44.4370332Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2022-11-23T01:34:44.4371183Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2022-11-23T01:34:44.4413897Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:44.4414213Z env: 2022-11-23T01:34:44.4414470Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:44.4414803Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:44.4415326Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:44.4415754Z ##[endgroup] 2022-11-23T01:34:45.2982884Z Collecting psutil==5.9.1 2022-11-23T01:34:45.5296380Z Installing collected packages: psutil 2022-11-23T01:34:45.6347125Z Successfully installed psutil-5.9.4 2022-11-23T01:34:46.4933188Z Collecting pynvml==11.4.1 2022-11-23T01:34:46.5808102Z Using cached https://files.pythonhosted.org/packages/cc/0a/47be6726fd13f1f4371fa858b506228ed12bc418c07ffcaa6c0f7ceedac0/pynvml-11.4.1-py3-none-any.whl 2022-11-23T01:34:46.5837770Z Installing collected packages: pynvml 2022-11-23T01:34:46.6275876Z Successfully installed pynvml-11.4.1 2022-11-23T01:34:46.6717787Z Prepare all required actions 2022-11-23T01:34:46.6718184Z Getting action download info 2022-11-23T01:34:46.9236945Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea) 2022-11-23T01:34:48.3862125Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7) 2022-11-23T01:34:49.2455143Z ##[group]Run ./.github/actions/download-build-artifacts 2022-11-23T01:34:49.2455398Z with: 2022-11-23T01:34:49.2455620Z name: linux-focal-rocm5.2-py3.8 2022-11-23T01:34:49.2455820Z env: 2022-11-23T01:34:49.2456020Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:49.2456276Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:49.2456679Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:49.2457022Z ##[endgroup] 2022-11-23T01:34:49.2484799Z ##[group]Run seemethere/download-artifact-s3@v4 2022-11-23T01:34:49.2485065Z with: 2022-11-23T01:34:49.2485292Z name: linux-focal-rocm5.2-py3.8 2022-11-23T01:34:49.2485526Z s3-bucket: gha-artifacts 2022-11-23T01:34:49.2485759Z region: us-east-1 2022-11-23T01:34:49.2485961Z env: 2022-11-23T01:34:49.2486173Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:49.2486437Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:34:49.2486830Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:49.2487193Z ##[endgroup] 2022-11-23T01:34:49.9648511Z Found 1 objects with prefix pytorch/pytorch/3528394938/linux-focal-rocm5.2-py3.8/ 2022-11-23T01:34:49.9650260Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:35:22.3583227Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:35:22.3587808Z 2022-11-23T01:35:22.3630668Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-11-23T01:35:22.3654345Z Artifact download has finished successfully 2022-11-23T01:35:22.3908363Z ##[group]Run unzip -o artifacts.zip 2022-11-23T01:35:22.3909171Z unzip -o artifacts.zip 2022-11-23T01:35:22.3968117Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:35:22.3968434Z env: 2022-11-23T01:35:22.3968695Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:35:22.3969035Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:35:22.3969530Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:35:22.3969981Z ##[endgroup] 2022-11-23T01:35:22.4058971Z Archive: artifacts.zip 2022-11-23T01:35:22.4061488Z creating: dist/ 2022-11-23T01:35:23.5769675Z inflating: dist/torch-1.14.0a0+git1cfd385-cp38-cp38-linux_x86_64.whl 2022-11-23T01:35:23.5770763Z creating: build/custom_test_artifacts/ 2022-11-23T01:35:23.5771944Z creating: build/custom_test_artifacts/custom-op-build/ 2022-11-23T01:35:23.5773279Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2022-11-23T01:35:23.5774749Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:35:23.5776211Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2022-11-23T01:35:23.5777751Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:35:23.5779342Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:35:23.5781353Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:35:23.5783257Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:35:23.5784911Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:35:23.5786433Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:35:23.5787980Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:35:23.5789755Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:35:23.5791443Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:35:23.5793141Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:35:23.5794833Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:35:23.5796547Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:35:23.5798263Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:35:23.5799779Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:35:23.5801454Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2022-11-23T01:35:23.5802951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2022-11-23T01:35:23.5804419Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2022-11-23T01:35:23.5806009Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2022-11-23T01:35:23.5807681Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2022-11-23T01:35:23.5809318Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2022-11-23T01:35:23.5810891Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2022-11-23T01:35:23.5812496Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2022-11-23T01:35:23.5814103Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2022-11-23T01:35:23.5815727Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2022-11-23T01:35:23.5817329Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2022-11-23T01:35:23.5818926Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2022-11-23T01:35:23.5820514Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2022-11-23T01:35:23.5901891Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2022-11-23T01:35:23.5903579Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2022-11-23T01:35:23.5905265Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2022-11-23T01:35:23.5907026Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2022-11-23T01:35:23.5908733Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2022-11-23T01:35:23.5910362Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2022-11-23T01:35:23.5912064Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2022-11-23T01:35:23.5914074Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2022-11-23T01:35:23.5915878Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2022-11-23T01:35:23.5917554Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2022-11-23T01:35:23.5919230Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2022-11-23T01:35:23.5921518Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2022-11-23T01:35:23.5989922Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2022-11-23T01:35:23.5991124Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:35:23.5992215Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:35:23.5993207Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2022-11-23T01:35:23.5994135Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2022-11-23T01:35:23.5995059Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2022-11-23T01:35:23.5995975Z inflating: build/custom_test_artifacts/custom-op-build/detect_rocm_version.cc 2022-11-23T01:35:23.5996854Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2022-11-23T01:35:23.5997693Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2022-11-23T01:35:23.5998549Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2022-11-23T01:35:23.6072771Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2022-11-23T01:35:23.6124123Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2022-11-23T01:35:23.6125547Z creating: build/custom_test_artifacts/jit-hook-build/ 2022-11-23T01:35:23.6126709Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2022-11-23T01:35:23.6128042Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:35:23.6129359Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2022-11-23T01:35:23.6130737Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:35:23.6132120Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:35:23.6133493Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:35:23.6134972Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:35:23.6136446Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:35:23.6137846Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:35:23.6139244Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:35:23.6140846Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:35:23.6142370Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:35:23.6143901Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:35:23.6145411Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:35:23.6146974Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:35:23.6148502Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:35:23.6150191Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:35:23.6151503Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2022-11-23T01:35:23.6152955Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2022-11-23T01:35:23.6154305Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2022-11-23T01:35:23.6155775Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2022-11-23T01:35:23.6157334Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2022-11-23T01:35:23.6158814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2022-11-23T01:35:23.6160527Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2022-11-23T01:35:23.6162047Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2022-11-23T01:35:23.6163517Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2022-11-23T01:35:23.6165022Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2022-11-23T01:35:23.6166487Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2022-11-23T01:35:23.6167975Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2022-11-23T01:35:23.6169517Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2022-11-23T01:35:23.6214050Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2022-11-23T01:35:23.6215785Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:35:23.6217348Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:35:23.6218771Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2022-11-23T01:35:23.6220114Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2022-11-23T01:35:23.6221433Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2022-11-23T01:35:23.6222749Z inflating: build/custom_test_artifacts/jit-hook-build/detect_rocm_version.cc 2022-11-23T01:35:23.6224021Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2022-11-23T01:35:23.6225196Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2022-11-23T01:35:23.6226421Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2022-11-23T01:35:23.6261008Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2022-11-23T01:35:23.6262445Z creating: build/custom_test_artifacts/custom-backend-build/ 2022-11-23T01:35:23.6263358Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2022-11-23T01:35:23.6264189Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:35:23.6265017Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2022-11-23T01:35:23.6265869Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:35:23.6266718Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:35:23.6267562Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:35:23.6268484Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:35:23.6269393Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:35:23.6270456Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:35:23.6271309Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:35:23.6272349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:35:23.6273305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:35:23.6274227Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:35:23.6275166Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:35:23.6276146Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:35:23.6277091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:35:23.6277927Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:35:23.6278748Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2022-11-23T01:35:23.6279585Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2022-11-23T01:35:23.6280552Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2022-11-23T01:35:23.6281443Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2022-11-23T01:35:23.6282415Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2022-11-23T01:35:23.6283344Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2022-11-23T01:35:23.6284245Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2022-11-23T01:35:23.6285171Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2022-11-23T01:35:23.6286088Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2022-11-23T01:35:23.6287013Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2022-11-23T01:35:23.6287923Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2022-11-23T01:35:23.6288820Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2022-11-23T01:35:23.6289788Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2022-11-23T01:35:23.6402223Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2022-11-23T01:35:23.6403213Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2022-11-23T01:35:23.6404243Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2022-11-23T01:35:23.6405225Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2022-11-23T01:35:23.6406179Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2022-11-23T01:35:23.6407115Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2022-11-23T01:35:23.6408064Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2022-11-23T01:35:23.6409015Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2022-11-23T01:35:23.6410152Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2022-11-23T01:35:23.6411103Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2022-11-23T01:35:23.6412133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2022-11-23T01:35:23.6422603Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2022-11-23T01:35:23.6502887Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2022-11-23T01:35:23.6504416Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:35:23.6505734Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:35:23.6507049Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2022-11-23T01:35:23.6508172Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2022-11-23T01:35:23.6509291Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2022-11-23T01:35:23.6510389Z inflating: build/custom_test_artifacts/custom-backend-build/detect_rocm_version.cc 2022-11-23T01:35:23.6511443Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2022-11-23T01:35:23.6512441Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2022-11-23T01:35:23.6513481Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2022-11-23T01:35:23.6576398Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2022-11-23T01:35:23.6616560Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2022-11-23T01:35:23.6617563Z creating: build/lib/ 2022-11-23T01:35:23.6618294Z inflating: build/lib/libclog.a 2022-11-23T01:35:23.6678542Z inflating: build/lib/libgtest.a 2022-11-23T01:35:23.6685728Z inflating: build/lib/libpthreadpool.a 2022-11-23T01:35:23.6759468Z inflating: build/lib/libbenchmark.a 2022-11-23T01:35:23.6845446Z inflating: build/lib/libprotobuf-lite.a 2022-11-23T01:35:23.6852116Z inflating: build/lib/libittnotify.a 2022-11-23T01:35:23.6880765Z inflating: build/lib/libtensorpipe_uv.a 2022-11-23T01:35:23.6943824Z inflating: build/lib/libasmjit.a 2022-11-23T01:35:23.7379923Z inflating: build/lib/libprotobuf.a 2022-11-23T01:35:23.7471213Z inflating: build/lib/libgloo.a 2022-11-23T01:35:23.7499520Z inflating: build/lib/libfmt.a 2022-11-23T01:35:23.7500412Z inflating: build/lib/libcaffe2_nvrtc.so 2022-11-23T01:35:23.7501231Z inflating: build/lib/libfoxi_loader.a 2022-11-23T01:35:23.7559386Z inflating: build/lib/libc10.so 2022-11-23T01:35:23.7560528Z inflating: build/lib/libtorch_global_deps.so 2022-11-23T01:35:23.7567047Z inflating: build/lib/libcpuinfo.a 2022-11-23T01:35:23.7569240Z inflating: build/lib/libnnpack_reference_layers.a 2022-11-23T01:35:23.7576545Z inflating: build/lib/libcpuinfo_internals.a 2022-11-23T01:35:23.7596230Z inflating: build/lib/libgmock.a 2022-11-23T01:35:23.7597099Z inflating: build/lib/libgtest_main.a 2022-11-23T01:35:23.7597943Z inflating: build/lib/libbenchmark_main.a 2022-11-23T01:35:24.5460963Z inflating: build/lib/libdnnl.a 2022-11-23T01:35:24.5946585Z inflating: build/lib/libprotoc.a 2022-11-23T01:35:24.6498440Z inflating: build/lib/libtensorpipe.a 2022-11-23T01:35:24.6863777Z inflating: build/lib/libgloo_hip.a 2022-11-23T01:35:24.6899665Z inflating: build/lib/libc10_hip.so 2022-11-23T01:35:24.6900538Z inflating: build/lib/libgmock_main.a 2022-11-23T01:35:24.8069301Z inflating: build/lib/libfbgemm.a 2022-11-23T01:35:24.8082164Z inflating: build/lib/libqnnpack.a 2022-11-23T01:35:24.9010270Z inflating: build/lib/libdnnl_graph.a 2022-11-23T01:35:24.9282810Z inflating: build/lib/libkineto.a 2022-11-23T01:35:24.9302325Z inflating: build/lib/libpytorch_qnnpack.a 2022-11-23T01:35:24.9340417Z inflating: build/lib/libcaffe2_protos.a 2022-11-23T01:35:24.9451520Z inflating: build/lib/libXNNPACK.a 2022-11-23T01:35:24.9491180Z inflating: build/lib/libonnx_proto.a 2022-11-23T01:35:25.0039762Z inflating: build/lib/libonnx.a 2022-11-23T01:35:25.0057870Z inflating: build/lib/libnnpack.a 2022-11-23T01:35:27.2154017Z inflating: build/lib/libtorch_cpu.so 2022-11-23T01:35:28.2864306Z inflating: build/lib/libtorch_hip.so 2022-11-23T01:35:28.2865215Z inflating: build/lib/libtorch.so 2022-11-23T01:35:28.2886447Z inflating: build/lib/libjitbackend_test.so 2022-11-23T01:35:28.2936502Z inflating: build/lib/libtorchbind_test.so 2022-11-23T01:35:28.2961739Z inflating: build/lib/libbackend_with_compiler.so 2022-11-23T01:35:28.2964295Z inflating: build/lib/libshm.so 2022-11-23T01:35:28.4516517Z inflating: build/lib/libtorch_python.so 2022-11-23T01:35:28.4550528Z inflating: build/lib/libnnapi_backend.so 2022-11-23T01:35:28.4551364Z creating: build/bin/ 2022-11-23T01:35:28.4552101Z creating: build/bin/CMakeFiles/ 2022-11-23T01:35:28.4553100Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:35:28.4554154Z extracting: build/bin/CMakeFiles/progress.marks 2022-11-23T01:35:28.4598360Z inflating: build/bin/hip_generator_test 2022-11-23T01:35:28.4599250Z inflating: build/bin/Makefile 2022-11-23T01:35:28.4646594Z inflating: build/bin/variant_test 2022-11-23T01:35:28.4647511Z inflating: build/bin/cmake_install.cmake 2022-11-23T01:35:28.4696544Z inflating: build/bin/undefined_tensor_test 2022-11-23T01:35:28.4697499Z inflating: build/bin/CTestTestfile.cmake 2022-11-23T01:35:28.4744877Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-11-23T01:35:28.4793867Z inflating: build/bin/c10_DeviceGuard_test 2022-11-23T01:35:28.4842234Z inflating: build/bin/c10_Device_test 2022-11-23T01:35:28.4898442Z inflating: build/bin/c10_DispatchKeySet_test 2022-11-23T01:35:28.4944895Z inflating: build/bin/c10_StreamGuard_test 2022-11-23T01:35:28.4992540Z inflating: build/bin/c10_SymInt_test 2022-11-23T01:35:28.5044602Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-11-23T01:35:28.5098633Z inflating: build/bin/c10_InlineStreamGuard_test 2022-11-23T01:35:28.5152269Z inflating: build/bin/c10_SizesAndStrides_test 2022-11-23T01:35:28.5198992Z inflating: build/bin/c10_Array_test 2022-11-23T01:35:28.5249668Z inflating: build/bin/c10_Bitset_test 2022-11-23T01:35:28.5296401Z inflating: build/bin/c10_ConstexprCrc_test 2022-11-23T01:35:28.5345594Z inflating: build/bin/c10_C++17_test 2022-11-23T01:35:28.5393324Z inflating: build/bin/c10_DeadlockDetection_test 2022-11-23T01:35:28.5441190Z inflating: build/bin/c10_Half_test 2022-11-23T01:35:28.5494573Z inflating: build/bin/c10_LeftRight_test 2022-11-23T01:35:28.5556108Z inflating: build/bin/c10_Metaprogramming_test 2022-11-23T01:35:28.5603832Z inflating: build/bin/c10_Synchronized_test 2022-11-23T01:35:28.5747174Z inflating: build/bin/c10_SmallVectorTest 2022-11-23T01:35:28.5800577Z inflating: build/bin/c10_ThreadLocal_test 2022-11-23T01:35:28.5849556Z inflating: build/bin/c10_TypeIndex_test 2022-11-23T01:35:28.5896315Z inflating: build/bin/c10_TypeTraits_test 2022-11-23T01:35:28.5944938Z inflating: build/bin/c10_TypeList_test 2022-11-23T01:35:28.5995168Z inflating: build/bin/c10_accumulate_test 2022-11-23T01:35:28.6048362Z inflating: build/bin/c10_bfloat16_test 2022-11-23T01:35:28.6101345Z inflating: build/bin/c10_complex_math_test 2022-11-23T01:35:28.6149517Z inflating: build/bin/c10_flags_test 2022-11-23T01:35:28.6199942Z inflating: build/bin/c10_exception_test 2022-11-23T01:35:28.6252651Z inflating: build/bin/c10_complex_test 2022-11-23T01:35:28.6357579Z inflating: build/bin/c10_either_test 2022-11-23T01:35:28.6406124Z inflating: build/bin/c10_irange_test 2022-11-23T01:35:28.6459697Z inflating: build/bin/c10_logging_test 2022-11-23T01:35:28.6615339Z inflating: build/bin/c10_intrusive_ptr_test 2022-11-23T01:35:28.6685957Z inflating: build/bin/c10_optional_test 2022-11-23T01:35:28.6737544Z inflating: build/bin/c10_registry_test 2022-11-23T01:35:28.6797052Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-11-23T01:35:28.6846018Z inflating: build/bin/c10_tempfile_test 2022-11-23T01:35:28.6901880Z inflating: build/bin/c10_string_view_test 2022-11-23T01:35:28.6955715Z inflating: build/bin/c10_typeid_test 2022-11-23T01:35:28.7003144Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-11-23T01:35:28.7431188Z inflating: build/bin/protoc-3.13.0.0 2022-11-23T01:35:28.7865889Z inflating: build/bin/protoc 2022-11-23T01:35:28.7913921Z inflating: build/bin/c10_hip_HIPTest 2022-11-23T01:35:28.8192265Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-11-23T01:35:28.8490795Z inflating: build/bin/vec_test_all_types_AVX512 2022-11-23T01:35:28.8801904Z inflating: build/bin/vec_test_all_types_AVX2 2022-11-23T01:35:28.8852937Z inflating: build/bin/HashStoreTest 2022-11-23T01:35:28.8903685Z inflating: build/bin/FileStoreTest 2022-11-23T01:35:28.8960997Z inflating: build/bin/TCPStoreTest 2022-11-23T01:35:28.8962327Z inflating: build/bin/example_allreduce 2022-11-23T01:35:28.9025198Z inflating: build/bin/ProcessGroupGlooTest 2022-11-23T01:35:28.9075739Z inflating: build/bin/Dimname_test 2022-11-23T01:35:28.9145462Z inflating: build/bin/Dict_test 2022-11-23T01:35:28.9200466Z inflating: build/bin/NamedTensor_test 2022-11-23T01:35:28.9261821Z inflating: build/bin/MaybeOwned_test 2022-11-23T01:35:28.9320753Z inflating: build/bin/static_runtime_bench 2022-11-23T01:35:28.9376381Z inflating: build/bin/apply_utils_test 2022-11-23T01:35:28.9435968Z inflating: build/bin/basic 2022-11-23T01:35:28.9491870Z inflating: build/bin/atest 2022-11-23T01:35:28.9545192Z inflating: build/bin/broadcast_test 2022-11-23T01:35:28.9600795Z inflating: build/bin/cpu_generator_test 2022-11-23T01:35:28.9651487Z inflating: build/bin/cpu_profiling_allocator_test 2022-11-23T01:35:28.9894019Z inflating: build/bin/static_runtime_test 2022-11-23T01:35:28.9942092Z inflating: build/bin/dispatch_key_set_test 2022-11-23T01:35:29.0027557Z inflating: build/bin/cpu_rng_test 2022-11-23T01:35:29.0075913Z inflating: build/bin/dlconvertor_test 2022-11-23T01:35:29.0131175Z inflating: build/bin/extension_backend_test 2022-11-23T01:35:29.0183856Z inflating: build/bin/half_test 2022-11-23T01:35:29.0231896Z inflating: build/bin/lazy_tensor_test 2022-11-23T01:35:29.0283689Z inflating: build/bin/math_kernel_test 2022-11-23T01:35:29.0373232Z inflating: build/bin/ivalue_test 2022-11-23T01:35:29.0424815Z inflating: build/bin/memory_format_test 2022-11-23T01:35:29.0475138Z inflating: build/bin/memory_overlapping_test 2022-11-23T01:35:29.0523642Z inflating: build/bin/operator_name_test 2022-11-23T01:35:29.0574055Z inflating: build/bin/mobile_memory_cleanup 2022-11-23T01:35:29.0628261Z inflating: build/bin/native_test 2022-11-23T01:35:29.0677493Z inflating: build/bin/operators_test 2022-11-23T01:35:29.0726817Z inflating: build/bin/packedtensoraccessor_test 2022-11-23T01:35:29.0781238Z inflating: build/bin/quantized_test 2022-11-23T01:35:29.0828853Z inflating: build/bin/reduce_ops_test 2022-11-23T01:35:29.0877557Z inflating: build/bin/reportMemoryUsage_test 2022-11-23T01:35:29.0940126Z inflating: build/bin/pow_test 2022-11-23T01:35:29.0994399Z inflating: build/bin/scalar_tensor_test 2022-11-23T01:35:29.1049299Z inflating: build/bin/scalar_test 2022-11-23T01:35:29.1099227Z inflating: build/bin/stride_properties_test 2022-11-23T01:35:29.1100634Z inflating: build/bin/thread_init_test 2022-11-23T01:35:29.1154900Z inflating: build/bin/type_ptr_test 2022-11-23T01:35:29.1229623Z inflating: build/bin/tensor_iterator_test 2022-11-23T01:35:29.1282393Z inflating: build/bin/test_parallel 2022-11-23T01:35:29.1340365Z inflating: build/bin/type_test 2022-11-23T01:35:29.1341292Z inflating: build/bin/verify_api_visibility 2022-11-23T01:35:29.1409688Z inflating: build/bin/vmap_test 2022-11-23T01:35:29.1460118Z inflating: build/bin/weakref_test 2022-11-23T01:35:29.1510044Z inflating: build/bin/wrapdim_test 2022-11-23T01:35:29.1558158Z inflating: build/bin/xla_tensor_test 2022-11-23T01:35:29.1659323Z inflating: build/bin/List_test 2022-11-23T01:35:29.1716603Z inflating: build/bin/IListRef_test 2022-11-23T01:35:29.1830736Z inflating: build/bin/kernel_function_legacy_test 2022-11-23T01:35:29.1893130Z inflating: build/bin/KernelFunction_test 2022-11-23T01:35:29.1983954Z inflating: build/bin/kernel_function_test 2022-11-23T01:35:29.2103275Z inflating: build/bin/kernel_lambda_legacy_test 2022-11-23T01:35:29.2200453Z inflating: build/bin/kernel_lambda_test 2022-11-23T01:35:29.2257537Z inflating: build/bin/kernel_stackbased_test 2022-11-23T01:35:29.2347872Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-11-23T01:35:29.2398135Z inflating: build/bin/CppSignature_test 2022-11-23T01:35:29.2445169Z inflating: build/bin/op_allowlist_test 2022-11-23T01:35:29.2499151Z inflating: build/bin/backend_fallback_test 2022-11-23T01:35:29.2545661Z inflating: build/bin/hip_complex_math_test 2022-11-23T01:35:29.2596564Z inflating: build/bin/inline_container_test 2022-11-23T01:35:29.2874773Z inflating: build/bin/op_registration_test 2022-11-23T01:35:29.2921522Z inflating: build/bin/hip_complex_test 2022-11-23T01:35:29.2972021Z inflating: build/bin/hip_apply_test 2022-11-23T01:35:29.3018601Z inflating: build/bin/hip_distributions_test 2022-11-23T01:35:29.3065838Z inflating: build/bin/hip_half_test 2022-11-23T01:35:29.3112537Z inflating: build/bin/hip_integer_divider_test 2022-11-23T01:35:29.3158999Z inflating: build/bin/hip_optional_test 2022-11-23T01:35:29.3205723Z inflating: build/bin/hip_packedtensoraccessor_test 2022-11-23T01:35:29.3254923Z inflating: build/bin/hip_dlconvertor_test 2022-11-23T01:35:29.3301934Z inflating: build/bin/hip_vectorized_test 2022-11-23T01:35:29.3317323Z inflating: build/bin/tutorial_tensorexpr 2022-11-23T01:35:29.3368988Z inflating: build/bin/test_dist_autograd 2022-11-23T01:35:29.3435155Z inflating: build/bin/test_cpp_rpc 2022-11-23T01:35:29.4190700Z inflating: build/bin/test_tensorexpr 2022-11-23T01:35:29.4192564Z inflating: build/bin/parallel_benchmark 2022-11-23T01:35:29.4257413Z inflating: build/bin/test_mobile_nnc 2022-11-23T01:35:29.4265012Z inflating: build/bin/aot_model_compiler_test 2022-11-23T01:35:29.4594593Z inflating: build/bin/test_lazy 2022-11-23T01:35:29.5664496Z inflating: build/bin/test_api 2022-11-23T01:35:29.5667122Z inflating: build/bin/torch_shm_manager 2022-11-23T01:35:29.6199216Z inflating: build/bin/test_jit 2022-11-23T01:35:29.6201625Z inflating: .pytorch-test-times.json 2022-11-23T01:35:29.6250744Z ##[group]Run df -H 2022-11-23T01:35:29.6251383Z df -H 2022-11-23T01:35:29.6313071Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:35:29.6313824Z env: 2022-11-23T01:35:29.6314409Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:35:29.6315210Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:35:29.6316491Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:35:29.6317584Z ##[endgroup] 2022-11-23T01:35:29.6401711Z Filesystem Size Used Avail Use% Mounted on 2022-11-23T01:35:29.6403747Z udev 271G 0 271G 0% /dev 2022-11-23T01:35:29.6405889Z tmpfs 55G 2.3M 55G 1% /run 2022-11-23T01:35:29.6407552Z /dev/mapper/ubuntu--server--x8664--vg-root 944G 361G 535G 41% / 2022-11-23T01:35:29.6408532Z tmpfs 271G 17k 271G 1% /dev/shm 2022-11-23T01:35:29.6409557Z tmpfs 5.3M 0 5.3M 0% /run/lock 2022-11-23T01:35:29.6410425Z tmpfs 271G 0 271G 0% /sys/fs/cgroup 2022-11-23T01:35:29.6411268Z /dev/sda1 754M 682M 34M 96% /boot 2022-11-23T01:35:29.6412143Z tmpfs 55G 13k 55G 1% /run/user/1120 2022-11-23T01:35:29.6413000Z tmpfs 55G 0 55G 0% /run/user/1000 2022-11-23T01:35:29.6461834Z ##[group]Run .github/scripts/parse_ref.py 2022-11-23T01:35:29.6462680Z .github/scripts/parse_ref.py 2022-11-23T01:35:29.6498315Z shell: /bin/bash -e {0} 2022-11-23T01:35:29.6498574Z env: 2022-11-23T01:35:29.6498820Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:35:29.6499129Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:35:29.6499642Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:35:29.6500084Z ##[endgroup] 2022-11-23T01:35:29.6947076Z ##[group]Run set -x 2022-11-23T01:35:29.6948008Z set -x 2022-11-23T01:35:29.6948583Z  2022-11-23T01:35:29.6949277Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-11-23T01:35:29.6950200Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-11-23T01:35:29.6951123Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-11-23T01:35:29.6951964Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-11-23T01:35:29.6952639Z else 2022-11-23T01:35:29.6953351Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:35:29.6954048Z fi 2022-11-23T01:35:29.6954603Z  2022-11-23T01:35:29.6955444Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2022-11-23T01:35:29.6956263Z  2022-11-23T01:35:29.6957003Z # sanitize the input commit message and PR body here: 2022-11-23T01:35:29.6957746Z # 2022-11-23T01:35:29.6958708Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2022-11-23T01:35:29.6960035Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2022-11-23T01:35:29.6961310Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2022-11-23T01:35:29.6962132Z PR_BODY="${PR_BODY//[$'\n\r']}" 2022-11-23T01:35:29.6962772Z  2022-11-23T01:35:29.6963690Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2022-11-23T01:35:29.6964677Z # wreak havoc internally 2022-11-23T01:35:29.6965521Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2022-11-23T01:35:29.6966367Z export PR_BODY="${PR_BODY//[\'\"]}" 2022-11-23T01:35:29.6967023Z  2022-11-23T01:35:29.6967826Z # detached container should get cleaned up by teardown_ec2_linux 2022-11-23T01:35:29.6968884Z # TODO: Stop building test binaries as part of the build phase 2022-11-23T01:35:29.6969847Z # Used for GPU_FLAG since that doesn't play nice 2022-11-23T01:35:29.6970700Z # shellcheck disable=SC2086,SC2090 2022-11-23T01:35:29.6971457Z container_name=$(docker run \ 2022-11-23T01:35:29.6972161Z  ${GPU_FLAG:-} \ 2022-11-23T01:35:29.6972850Z  -e BUILD_ENVIRONMENT \ 2022-11-23T01:35:29.6973540Z  -e PR_NUMBER \ 2022-11-23T01:35:29.6974218Z  -e GITHUB_ACTIONS \ 2022-11-23T01:35:29.6974869Z  -e BRANCH \ 2022-11-23T01:35:29.6975466Z  -e SHA1 \ 2022-11-23T01:35:29.6976118Z  -e AWS_DEFAULT_REGION \ 2022-11-23T01:35:29.6977036Z  -e IN_WHEEL_TEST \ 2022-11-23T01:35:29.6977710Z  -e SHARD_NUMBER \ 2022-11-23T01:35:29.6978376Z  -e TEST_CONFIG \ 2022-11-23T01:35:29.6979047Z  -e NUM_TEST_SHARDS \ 2022-11-23T01:35:29.6979708Z  -e PR_BODY \ 2022-11-23T01:35:29.6980348Z  -e COMMIT_MESSAGES \ 2022-11-23T01:35:29.6981103Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-11-23T01:35:29.6981906Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2022-11-23T01:35:29.6982731Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-11-23T01:35:29.6983477Z  -e SCCACHE_BUCKET \ 2022-11-23T01:35:29.6984237Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-11-23T01:35:29.6985061Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2022-11-23T01:35:29.6985903Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2022-11-23T01:35:29.6986809Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-11-23T01:35:29.6987642Z  --ulimit stack=10485760:83886080 \ 2022-11-23T01:35:29.6988461Z  --security-opt seccomp=unconfined \ 2022-11-23T01:35:29.6989254Z  --cap-add=SYS_PTRACE \ 2022-11-23T01:35:29.6989956Z  --shm-size="8g" \ 2022-11-23T01:35:29.6990563Z  --tty \ 2022-11-23T01:35:29.6991164Z  --detach \ 2022-11-23T01:35:29.6992024Z  --name="${container_name}" \ 2022-11-23T01:35:29.6992745Z  --user jenkins \ 2022-11-23T01:35:29.6993589Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-11-23T01:35:29.6994481Z  -w /var/lib/jenkins/workspace \ 2022-11-23T01:35:29.6995182Z  "${DOCKER_IMAGE}" 2022-11-23T01:35:29.6995786Z ) 2022-11-23T01:35:29.6996464Z # save container name for later step 2022-11-23T01:35:29.6997360Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2022-11-23T01:35:29.6998604Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2022-11-23T01:35:29.7000128Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2022-11-23T01:35:29.7025269Z shell: /bin/bash -e {0} 2022-11-23T01:35:29.7025508Z env: 2022-11-23T01:35:29.7025756Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:35:29.7026075Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T01:35:29.7026586Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:35:29.7027091Z BUILD_ENVIRONMENT: linux-focal-rocm5.2-py3.8 2022-11-23T01:35:29.7027395Z PR_NUMBER: 2022-11-23T01:35:29.7027619Z BRANCH: master 2022-11-23T01:35:29.7027905Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:35:29.7028210Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T01:35:29.7028507Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T01:35:29.7028802Z TEST_CONFIG: distributed 2022-11-23T01:35:29.7029059Z SHARD_NUMBER: 2 2022-11-23T01:35:29.7029288Z NUM_TEST_SHARDS: 2 2022-11-23T01:35:29.7029531Z PR_BODY: 2022-11-23T01:35:29.7029852Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-11-23T01:35:29.7030426Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:35:29.7031026Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:35:29.7031418Z PYTORCH_JIT_ENABLE_NVFUSER: 1 2022-11-23T01:35:29.7031706Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2022-11-23T01:35:29.7032017Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 1 2022-11-23T01:35:29.7032293Z ##[endgroup] 2022-11-23T01:35:29.7080934Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-11-23T01:35:29.7082462Z + [[ linux-focal-rocm5.2-py3.8 == *onnx* ]] 2022-11-23T01:35:29.7083314Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:35:29.7087584Z ++ git cherry -v origin/master 2022-11-23T01:35:29.7126357Z + COMMIT_MESSAGES= 2022-11-23T01:35:29.7127336Z + COMMIT_MESSAGES= 2022-11-23T01:35:29.7128082Z + PR_BODY= 2022-11-23T01:35:29.7128754Z + export COMMIT_MESSAGES= 2022-11-23T01:35:29.7129404Z + COMMIT_MESSAGES= 2022-11-23T01:35:29.7130006Z + export PR_BODY= 2022-11-23T01:35:29.7130593Z + PR_BODY= 2022-11-23T01:35:29.7150326Z +++ nproc --ignore=2 2022-11-23T01:35:29.7181749Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e MAX_JOBS=54 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528394938 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:35:32.3787891Z + container_name=ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T01:35:32.3789461Z + echo CONTAINER_NAME=ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T01:35:32.3792238Z + docker exec -t ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .jenkins/pytorch/test.sh' 2022-11-23T01:35:38.7589001Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp38-cp38-linux_x86_64.whl 2022-11-23T01:35:39.2412273Z Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3) 2022-11-23T01:35:39.2414211Z Requirement already satisfied: sympy in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (1.11.1) 2022-11-23T01:35:39.2416128Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (4.4.0) 2022-11-23T01:35:39.2565152Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1) 2022-11-23T01:35:39.8836497Z Installing collected packages: torch 2022-11-23T01:35:45.9768466Z Successfully installed torch-1.14.0a0+git1cfd385 2022-11-23T01:35:46.0398928Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-11-23T01:35:46.0696604Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.8/site-packages/torch 2022-11-23T01:35:46.0697963Z + TORCH_BIN_DIR=/opt/conda/lib/python3.8/site-packages/torch/bin 2022-11-23T01:35:46.0699216Z + TORCH_LIB_DIR=/opt/conda/lib/python3.8/site-packages/torch/lib 2022-11-23T01:35:46.0700516Z + TORCH_TEST_DIR=/opt/conda/lib/python3.8/site-packages/torch/test 2022-11-23T01:35:46.0701331Z + BUILD_DIR=build 2022-11-23T01:35:46.0701984Z + BUILD_RENAMED_DIR=build_renamed 2022-11-23T01:35:46.0702675Z + BUILD_BIN_DIR=build/bin 2022-11-23T01:35:46.0703335Z + export VALGRIND=ON 2022-11-23T01:35:46.0703947Z + VALGRIND=ON 2022-11-23T01:35:46.0704903Z + [[ linux-focal-rocm5.2-py3.8 == *clang9* ]] 2022-11-23T01:35:46.0705945Z + [[ linux-focal-rocm5.2-py3.8 != *bazel* ]] 2022-11-23T01:35:46.0707835Z ++ realpath build/custom_test_artifacts 2022-11-23T01:35:46.0731525Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2022-11-23T01:35:46.0737465Z ++ dirname .jenkins/pytorch/test.sh 2022-11-23T01:35:46.0759932Z + source .jenkins/pytorch/common.sh 2022-11-23T01:35:46.0769721Z +++ dirname .jenkins/pytorch/common.sh 2022-11-23T01:35:46.0789205Z ++ source .jenkins/pytorch/common_utils.sh 2022-11-23T01:35:46.0790305Z +++ declare -f -t trap_add 2022-11-23T01:35:46.0810278Z ++ set -ex 2022-11-23T01:35:46.0811341Z ++ [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:35:46.0812124Z ++ unset HIP_PLATFORM 2022-11-23T01:35:46.0812844Z ++ export PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:35:46.0813573Z ++ PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:35:46.0814277Z ++ export HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:35:46.0814932Z ++ HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:35:46.0815665Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:35:46.0816406Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:35:46.0817088Z ++ BUILD_TEST_LIBTORCH=0 2022-11-23T01:35:46.0817951Z + echo 'Environment variables' 2022-11-23T01:35:46.0818617Z Environment variables 2022-11-23T01:35:46.0819211Z + env 2022-11-23T01:35:46.0826616Z INSTALLED_DB=yes 2022-11-23T01:35:46.0827850Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:35:46.0828839Z BUILD_ENVIRONMENT=linux-focal-rocm5.2-py3.8 2022-11-23T01:35:46.0829486Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2022-11-23T01:35:46.0830022Z HOSTNAME=ff81ba00bb81 2022-11-23T01:35:46.0831175Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_e32f726f-bdea-4a5d-9cb6-42960bc32442 2022-11-23T01:35:46.0864916Z GITHUB_ACTION=__self 2022-11-23T01:35:46.0865518Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2022-11-23T01:35:46.0866086Z GITHUB_RUN_NUMBER=3445 2022-11-23T01:35:46.0866934Z TEST_CONFIG=distributed 2022-11-23T01:35:46.0867535Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2022-11-23T01:35:46.0868119Z GITHUB_REF_TYPE=branch 2022-11-23T01:35:46.0872532Z *** 2022-11-23T01:35:46.0873154Z GITHUB_ACTIONS=true 2022-11-23T01:35:46.0873711Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:35:46.0874375Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:35:46.0874986Z GITHUB_REF=refs/heads/master 2022-11-23T01:35:46.0875494Z SHARD_NUMBER=2 2022-11-23T01:35:46.0875996Z GITHUB_REF_PROTECTED=true 2022-11-23T01:35:46.0876514Z HOME=/var/lib/jenkins 2022-11-23T01:35:46.0877074Z GITHUB_API_URL=https://api.github.com 2022-11-23T01:35:46.0877726Z PYTORCH_TEST_RERUN_DISABLED_TESTS=1 2022-11-23T01:35:46.0878394Z LANG=C.UTF-8 2022-11-23T01:35:46.0878895Z PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:35:46.0879403Z NUM_TEST_SHARDS=2 2022-11-23T01:35:46.0880702Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_e32f726f-bdea-4a5d-9cb6-42960bc32442 2022-11-23T01:35:46.0881561Z MAGMA_HOME=/opt/rocm/magma 2022-11-23T01:35:46.0882103Z PYTORCH_RETRY_TEST_CASES=1 2022-11-23T01:35:46.0883273Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_e32f726f-bdea-4a5d-9cb6-42960bc32442 2022-11-23T01:35:46.0884079Z HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:35:46.0885046Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2022-11-23T01:35:46.0885768Z GITHUB_EVENT_NAME=schedule 2022-11-23T01:35:46.0886282Z GITHUB_RUN_ID=3528394938 2022-11-23T01:35:46.0887498Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_e32f726f-bdea-4a5d-9cb6-42960bc32442 2022-11-23T01:35:46.0888394Z GITHUB_ACTOR=pytorchmergebot 2022-11-23T01:35:46.0888902Z PR_NUMBER= 2022-11-23T01:35:46.0889364Z GITHUB_RUN_ATTEMPT=1 2022-11-23T01:35:46.0889820Z VALGRIND=ON 2022-11-23T01:35:46.0890432Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2022-11-23T01:35:46.0891017Z TERM=xterm 2022-11-23T01:35:46.0891477Z INSTALLED_VISION=yes 2022-11-23T01:35:46.0891951Z BRANCH=master 2022-11-23T01:35:46.0892942Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2022-11-23T01:35:46.0893947Z GITHUB_SERVER_URL=https://github.com 2022-11-23T01:35:46.0894532Z PYTORCH_ROCM_ARCH=gfx906 2022-11-23T01:35:46.0895002Z SHLVL=1 2022-11-23T01:35:46.0895415Z MAX_JOBS=54 2022-11-23T01:35:46.0895862Z COMMIT_MESSAGES= 2022-11-23T01:35:46.0896329Z GITHUB_REF_NAME=master 2022-11-23T01:35:46.0897272Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:35:46.0897996Z GITHUB_JOB=test 2022-11-23T01:35:46.0898750Z GITHUB_REPOSITORY=pytorch/pytorch 2022-11-23T01:35:46.0899339Z LC_ALL=C.UTF-8 2022-11-23T01:35:46.0899817Z GITHUB_RETENTION_DAYS=90 2022-11-23T01:35:46.0900356Z GITHUB_ACTION_REPOSITORY= 2022-11-23T01:35:46.0901360Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:35:46.0902254Z GITHUB_BASE_REF= 2022-11-23T01:35:46.0902709Z CI=true 2022-11-23T01:35:46.0903177Z HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:35:46.0903723Z GITHUB_REPOSITORY_OWNER=pytorch 2022-11-23T01:35:46.0904271Z INSTALLED_PROTOBUF=yes 2022-11-23T01:35:46.0904764Z GITHUB_HEAD_REF= 2022-11-23T01:35:46.0905237Z GITHUB_ACTION_REF= 2022-11-23T01:35:46.0906029Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2022-11-23T01:35:46.0906687Z GITHUB_WORKFLOW=periodic 2022-11-23T01:35:46.0907219Z DEBIAN_FRONTEND=noninteractive 2022-11-23T01:35:46.0908423Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_e32f726f-bdea-4a5d-9cb6-42960bc32442 2022-11-23T01:35:46.0909254Z OLDPWD=/var/lib/jenkins 2022-11-23T01:35:46.0909726Z PR_BODY= 2022-11-23T01:35:46.0910156Z _=/usr/bin/env 2022-11-23T01:35:46.0910721Z + echo 'Testing pytorch' 2022-11-23T01:35:46.0911220Z Testing pytorch 2022-11-23T01:35:46.0911756Z + export LANG=C.UTF-8 2022-11-23T01:35:46.0912403Z + LANG=C.UTF-8 2022-11-23T01:35:46.0912856Z + PR_NUMBER= 2022-11-23T01:35:46.0913362Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-11-23T01:35:46.0913954Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2022-11-23T01:35:46.0914752Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:35:46.0915375Z + export HIP_VISIBLE_DEVICES=0,1 2022-11-23T01:35:46.0915921Z + HIP_VISIBLE_DEVICES=0,1 2022-11-23T01:35:46.0916452Z + [[ distributed == \s\l\o\w ]] 2022-11-23T01:35:46.0917239Z + [[ linux-focal-rocm5.2-py3.8 == *slow-gradcheck* ]] 2022-11-23T01:35:46.0918091Z + [[ linux-focal-rocm5.2-py3.8 == *cuda* ]] 2022-11-23T01:35:46.0918897Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:35:46.0919555Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-11-23T01:35:46.0920301Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-11-23T01:35:46.0920897Z + [[ distributed == *crossref* ]] 2022-11-23T01:35:46.0921430Z + [[ distributed == *dynamo* ]] 2022-11-23T01:35:46.0921982Z + [[ distributed == *inductor* ]] 2022-11-23T01:35:46.0922729Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:35:46.0923276Z + rocminfo 2022-11-23T01:35:46.1102175Z ROCk module is loaded 2022-11-23T01:35:46.1670616Z ===================== 2022-11-23T01:35:46.1671356Z HSA System Attributes 2022-11-23T01:35:46.1672008Z ===================== 2022-11-23T01:35:46.1672648Z Runtime Version: 1.1 2022-11-23T01:35:46.1673406Z System Timestamp Freq.: 1000.000000MHz 2022-11-23T01:35:46.1674498Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2022-11-23T01:35:46.1675492Z Machine Model: LARGE 2022-11-23T01:35:46.1676376Z System Endianness: LITTLE 2022-11-23T01:35:46.1676884Z 2022-11-23T01:35:46.1677117Z ========== 2022-11-23T01:35:46.1677711Z HSA Agents 2022-11-23T01:35:46.1678326Z ========== 2022-11-23T01:35:46.1678908Z ******* 2022-11-23T01:35:46.1679494Z Agent 1 2022-11-23T01:35:46.1680094Z ******* 2022-11-23T01:35:46.1681123Z Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:35:46.1682351Z Uuid: CPU-XX 2022-11-23T01:35:46.1683246Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:35:46.1684137Z Vendor Name: CPU 2022-11-23T01:35:46.1684968Z Feature: None specified 2022-11-23T01:35:46.1685807Z Profile: FULL_PROFILE 2022-11-23T01:35:46.1686979Z Float Round Mode: NEAR 2022-11-23T01:35:46.1687770Z Max Queue Number: 0(0x0) 2022-11-23T01:35:46.1688567Z Queue Min Size: 0(0x0) 2022-11-23T01:35:46.1689368Z Queue Max Size: 0(0x0) 2022-11-23T01:35:46.1690154Z Queue Type: MULTI 2022-11-23T01:35:46.1690925Z Node: 0 2022-11-23T01:35:46.1691671Z Device Type: CPU 2022-11-23T01:35:46.1692349Z Cache Info: 2022-11-23T01:35:46.1693087Z L1: 32768(0x8000) KB 2022-11-23T01:35:46.1693872Z Chip ID: 0(0x0) 2022-11-23T01:35:46.1694650Z Cacheline Size: 64(0x40) 2022-11-23T01:35:46.1695462Z Max Clock Freq. (MHz): 3700 2022-11-23T01:35:46.1696230Z BDFID: 0 2022-11-23T01:35:46.1697002Z Internal Node ID: 0 2022-11-23T01:35:46.1697751Z Compute Unit: 28 2022-11-23T01:35:46.1698639Z SIMDs per CU: 0 2022-11-23T01:35:46.1699427Z Shader Engines: 0 2022-11-23T01:35:46.1700238Z Shader Arrs. per Eng.: 0 2022-11-23T01:35:46.1701074Z WatchPts on Addr. Ranges:1 2022-11-23T01:35:46.1701811Z Features: None 2022-11-23T01:35:46.1702452Z Pool Info: 2022-11-23T01:35:46.1703112Z Pool 1 2022-11-23T01:35:46.1704093Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:46.1704965Z Size: 263817324(0xfb9886c) KB 2022-11-23T01:35:46.1705813Z Allocatable: TRUE 2022-11-23T01:35:46.1706645Z Alloc Granule: 4KB 2022-11-23T01:35:46.1707448Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1708306Z Accessible by all: TRUE 2022-11-23T01:35:46.1709036Z Pool 2 2022-11-23T01:35:46.1709847Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:35:46.1710723Z Size: 263817324(0xfb9886c) KB 2022-11-23T01:35:46.1711542Z Allocatable: TRUE 2022-11-23T01:35:46.1712341Z Alloc Granule: 4KB 2022-11-23T01:35:46.1713155Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1713988Z Accessible by all: TRUE 2022-11-23T01:35:46.1714724Z Pool 3 2022-11-23T01:35:46.1715511Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:46.1716362Z Size: 263817324(0xfb9886c) KB 2022-11-23T01:35:46.1717155Z Allocatable: TRUE 2022-11-23T01:35:46.1717982Z Alloc Granule: 4KB 2022-11-23T01:35:46.1718787Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1719623Z Accessible by all: TRUE 2022-11-23T01:35:46.1720515Z ISA Info: 2022-11-23T01:35:46.1721133Z ******* 2022-11-23T01:35:46.1721712Z Agent 2 2022-11-23T01:35:46.1722315Z ******* 2022-11-23T01:35:46.1723072Z Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:35:46.1724345Z Uuid: CPU-XX 2022-11-23T01:35:46.1725223Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:35:46.1726117Z Vendor Name: CPU 2022-11-23T01:35:46.1726950Z Feature: None specified 2022-11-23T01:35:46.1727774Z Profile: FULL_PROFILE 2022-11-23T01:35:46.1728588Z Float Round Mode: NEAR 2022-11-23T01:35:46.1729418Z Max Queue Number: 0(0x0) 2022-11-23T01:35:46.1730231Z Queue Min Size: 0(0x0) 2022-11-23T01:35:46.1731026Z Queue Max Size: 0(0x0) 2022-11-23T01:35:46.1731813Z Queue Type: MULTI 2022-11-23T01:35:46.1732555Z Node: 1 2022-11-23T01:35:46.1733319Z Device Type: CPU 2022-11-23T01:35:46.1734027Z Cache Info: 2022-11-23T01:35:46.1734758Z L1: 32768(0x8000) KB 2022-11-23T01:35:46.1735535Z Chip ID: 0(0x0) 2022-11-23T01:35:46.1736435Z Cacheline Size: 64(0x40) 2022-11-23T01:35:46.1737413Z Max Clock Freq. (MHz): 3700 2022-11-23T01:35:46.1738178Z BDFID: 0 2022-11-23T01:35:46.1738938Z Internal Node ID: 1 2022-11-23T01:35:46.1739697Z Compute Unit: 28 2022-11-23T01:35:46.1740478Z SIMDs per CU: 0 2022-11-23T01:35:46.1741257Z Shader Engines: 0 2022-11-23T01:35:46.1742055Z Shader Arrs. per Eng.: 0 2022-11-23T01:35:46.1742887Z WatchPts on Addr. Ranges:1 2022-11-23T01:35:46.1743626Z Features: None 2022-11-23T01:35:46.1744290Z Pool Info: 2022-11-23T01:35:46.1744918Z Pool 1 2022-11-23T01:35:46.1745719Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:46.1746591Z Size: 264203864(0xfbf6e58) KB 2022-11-23T01:35:46.1747418Z Allocatable: TRUE 2022-11-23T01:35:46.1748252Z Alloc Granule: 4KB 2022-11-23T01:35:46.1749068Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1749874Z Accessible by all: TRUE 2022-11-23T01:35:46.1750610Z Pool 2 2022-11-23T01:35:46.1751416Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:35:46.1752292Z Size: 264203864(0xfbf6e58) KB 2022-11-23T01:35:46.1753128Z Allocatable: TRUE 2022-11-23T01:35:46.1753949Z Alloc Granule: 4KB 2022-11-23T01:35:46.1754747Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1755587Z Accessible by all: TRUE 2022-11-23T01:35:46.1756310Z Pool 3 2022-11-23T01:35:46.1757098Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:46.1757951Z Size: 264203864(0xfbf6e58) KB 2022-11-23T01:35:46.1758777Z Allocatable: TRUE 2022-11-23T01:35:46.1759561Z Alloc Granule: 4KB 2022-11-23T01:35:46.1760479Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1761424Z Accessible by all: TRUE 2022-11-23T01:35:46.1762154Z ISA Info: 2022-11-23T01:35:46.1762772Z ******* 2022-11-23T01:35:46.1763378Z Agent 3 2022-11-23T01:35:46.1763951Z ******* 2022-11-23T01:35:46.1764648Z Name: gfx906 2022-11-23T01:35:46.1765752Z Uuid: GPU-d422688172dc76e6 2022-11-23T01:35:46.1766567Z Marketing Name: 2022-11-23T01:35:46.1767363Z Vendor Name: AMD 2022-11-23T01:35:46.1768186Z Feature: KERNEL_DISPATCH 2022-11-23T01:35:46.1768996Z Profile: BASE_PROFILE 2022-11-23T01:35:46.1769819Z Float Round Mode: NEAR 2022-11-23T01:35:46.1770611Z Max Queue Number: 128(0x80) 2022-11-23T01:35:46.1771410Z Queue Min Size: 64(0x40) 2022-11-23T01:35:46.1772198Z Queue Max Size: 131072(0x20000) 2022-11-23T01:35:46.1772982Z Queue Type: MULTI 2022-11-23T01:35:46.1773850Z Node: 2 2022-11-23T01:35:46.1774585Z Device Type: GPU 2022-11-23T01:35:46.1775271Z Cache Info: 2022-11-23T01:35:46.1776001Z L1: 16(0x10) KB 2022-11-23T01:35:46.1776788Z Chip ID: 26273(0x66a1) 2022-11-23T01:35:46.1777574Z Cacheline Size: 64(0x40) 2022-11-23T01:35:46.1778372Z Max Clock Freq. (MHz): 1725 2022-11-23T01:35:46.1779098Z BDFID: 16128 2022-11-23T01:35:46.1779860Z Internal Node ID: 2 2022-11-23T01:35:46.1780638Z Compute Unit: 60 2022-11-23T01:35:46.1781401Z SIMDs per CU: 4 2022-11-23T01:35:46.1782183Z Shader Engines: 4 2022-11-23T01:35:46.1782988Z Shader Arrs. per Eng.: 1 2022-11-23T01:35:46.1783801Z WatchPts on Addr. Ranges:4 2022-11-23T01:35:46.1784606Z Features: KERNEL_DISPATCH 2022-11-23T01:35:46.1785416Z Fast F16 Operation: TRUE 2022-11-23T01:35:46.1786213Z Wavefront Size: 64(0x40) 2022-11-23T01:35:46.1787020Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:35:46.1787804Z Workgroup Max Size per Dimension: 2022-11-23T01:35:46.1788568Z x 1024(0x400) 2022-11-23T01:35:46.1789328Z y 1024(0x400) 2022-11-23T01:35:46.1790042Z z 1024(0x400) 2022-11-23T01:35:46.1790830Z Max Waves Per CU: 40(0x28) 2022-11-23T01:35:46.1791924Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:35:46.1792768Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:35:46.1793542Z Grid Max Size per Dimension: 2022-11-23T01:35:46.1794284Z x 4294967295(0xffffffff) 2022-11-23T01:35:46.1795061Z y 4294967295(0xffffffff) 2022-11-23T01:35:46.1795844Z z 4294967295(0xffffffff) 2022-11-23T01:35:46.1796658Z Max fbarriers/Workgrp: 32 2022-11-23T01:35:46.1797383Z Pool Info: 2022-11-23T01:35:46.1798025Z Pool 1 2022-11-23T01:35:46.1798994Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:46.1799867Z Size: 16760832(0xffc000) KB 2022-11-23T01:35:46.1800848Z Allocatable: TRUE 2022-11-23T01:35:46.1801697Z Alloc Granule: 4KB 2022-11-23T01:35:46.1802527Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1803383Z Accessible by all: FALSE 2022-11-23T01:35:46.1804084Z Pool 2 2022-11-23T01:35:46.1805009Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:46.1805365Z Size: 16760832(0xffc000) KB 2022-11-23T01:35:46.1805697Z Allocatable: TRUE 2022-11-23T01:35:46.1806032Z Alloc Granule: 4KB 2022-11-23T01:35:46.1806374Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1806698Z Accessible by all: FALSE 2022-11-23T01:35:46.1806995Z Pool 3 2022-11-23T01:35:46.1807300Z Segment: GROUP 2022-11-23T01:35:46.1807683Z Size: 64(0x40) KB 2022-11-23T01:35:46.1808020Z Allocatable: FALSE 2022-11-23T01:35:46.1808360Z Alloc Granule: 0KB 2022-11-23T01:35:46.1808692Z Alloc Alignment: 0KB 2022-11-23T01:35:46.1809011Z Accessible by all: FALSE 2022-11-23T01:35:46.1809315Z ISA Info: 2022-11-23T01:35:46.1809584Z ISA 1 2022-11-23T01:35:46.1810050Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:35:46.1810459Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:35:46.1810827Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:35:46.1811162Z Default Rounding Mode: NEAR 2022-11-23T01:35:46.1811520Z Default Rounding Mode: NEAR 2022-11-23T01:35:46.1811853Z Fast f16: TRUE 2022-11-23T01:35:46.1812182Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:35:46.1812512Z Workgroup Max Size per Dimension: 2022-11-23T01:35:46.1812846Z x 1024(0x400) 2022-11-23T01:35:46.1813144Z y 1024(0x400) 2022-11-23T01:35:46.1813449Z z 1024(0x400) 2022-11-23T01:35:46.1813782Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:35:46.1814109Z Grid Max Size per Dimension: 2022-11-23T01:35:46.1814432Z x 4294967295(0xffffffff) 2022-11-23T01:35:46.1814757Z y 4294967295(0xffffffff) 2022-11-23T01:35:46.1815068Z z 4294967295(0xffffffff) 2022-11-23T01:35:46.1815405Z FBarrier Max Size: 32 2022-11-23T01:35:46.1815698Z ******* 2022-11-23T01:35:46.1815954Z Agent 4 2022-11-23T01:35:46.1816203Z ******* 2022-11-23T01:35:46.1816490Z Name: gfx906 2022-11-23T01:35:46.1816897Z Uuid: GPU-17e038e172fd5d44 2022-11-23T01:35:46.1817228Z Marketing Name: 2022-11-23T01:35:46.1817546Z Vendor Name: AMD 2022-11-23T01:35:46.1817941Z Feature: KERNEL_DISPATCH 2022-11-23T01:35:46.1818286Z Profile: BASE_PROFILE 2022-11-23T01:35:46.1818622Z Float Round Mode: NEAR 2022-11-23T01:35:46.1818952Z Max Queue Number: 128(0x80) 2022-11-23T01:35:46.1819263Z Queue Min Size: 64(0x40) 2022-11-23T01:35:46.1819592Z Queue Max Size: 131072(0x20000) 2022-11-23T01:35:46.1819925Z Queue Type: MULTI 2022-11-23T01:35:46.1820246Z Node: 3 2022-11-23T01:35:46.1820562Z Device Type: GPU 2022-11-23T01:35:46.1820848Z Cache Info: 2022-11-23T01:35:46.1821132Z L1: 16(0x10) KB 2022-11-23T01:35:46.1821456Z Chip ID: 26273(0x66a1) 2022-11-23T01:35:46.1821786Z Cacheline Size: 64(0x40) 2022-11-23T01:35:46.1822112Z Max Clock Freq. (MHz): 1725 2022-11-23T01:35:46.1822428Z BDFID: 17152 2022-11-23T01:35:46.1822792Z Internal Node ID: 3 2022-11-23T01:35:46.1823095Z Compute Unit: 60 2022-11-23T01:35:46.1823406Z SIMDs per CU: 4 2022-11-23T01:35:46.1823727Z Shader Engines: 4 2022-11-23T01:35:46.1824053Z Shader Arrs. per Eng.: 1 2022-11-23T01:35:46.1824393Z WatchPts on Addr. Ranges:4 2022-11-23T01:35:46.1824720Z Features: KERNEL_DISPATCH 2022-11-23T01:35:46.1825036Z Fast F16 Operation: TRUE 2022-11-23T01:35:46.1825375Z Wavefront Size: 64(0x40) 2022-11-23T01:35:46.1825714Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:35:46.1826041Z Workgroup Max Size per Dimension: 2022-11-23T01:35:46.1826365Z x 1024(0x400) 2022-11-23T01:35:46.1826675Z y 1024(0x400) 2022-11-23T01:35:46.1826977Z z 1024(0x400) 2022-11-23T01:35:46.1827280Z Max Waves Per CU: 40(0x28) 2022-11-23T01:35:46.1827714Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:35:46.1828062Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:35:46.1828376Z Grid Max Size per Dimension: 2022-11-23T01:35:46.1828693Z x 4294967295(0xffffffff) 2022-11-23T01:35:46.1829011Z y 4294967295(0xffffffff) 2022-11-23T01:35:46.1829318Z z 4294967295(0xffffffff) 2022-11-23T01:35:46.1829644Z Max fbarriers/Workgrp: 32 2022-11-23T01:35:46.1829941Z Pool Info: 2022-11-23T01:35:46.1830210Z Pool 1 2022-11-23T01:35:46.1830541Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:46.1830903Z Size: 16760832(0xffc000) KB 2022-11-23T01:35:46.1831223Z Allocatable: TRUE 2022-11-23T01:35:46.1831563Z Alloc Granule: 4KB 2022-11-23T01:35:46.1831900Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1832244Z Accessible by all: FALSE 2022-11-23T01:35:46.1832543Z Pool 2 2022-11-23T01:35:46.1832866Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:46.1833242Z Size: 16760832(0xffc000) KB 2022-11-23T01:35:46.1833574Z Allocatable: TRUE 2022-11-23T01:35:46.1833911Z Alloc Granule: 4KB 2022-11-23T01:35:46.1834248Z Alloc Alignment: 4KB 2022-11-23T01:35:46.1834594Z Accessible by all: FALSE 2022-11-23T01:35:46.1834892Z Pool 3 2022-11-23T01:35:46.1835179Z Segment: GROUP 2022-11-23T01:35:46.1835505Z Size: 64(0x40) KB 2022-11-23T01:35:46.1835837Z Allocatable: FALSE 2022-11-23T01:35:46.1836176Z Alloc Granule: 0KB 2022-11-23T01:35:46.1836504Z Alloc Alignment: 0KB 2022-11-23T01:35:46.1836849Z Accessible by all: FALSE 2022-11-23T01:35:46.1837147Z ISA Info: 2022-11-23T01:35:46.1837394Z ISA 1 2022-11-23T01:35:46.1837850Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:35:46.1838300Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:35:46.1838668Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:35:46.1839020Z Default Rounding Mode: NEAR 2022-11-23T01:35:46.1839368Z Default Rounding Mode: NEAR 2022-11-23T01:35:46.1839682Z Fast f16: TRUE 2022-11-23T01:35:46.1840014Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:35:46.1840408Z Workgroup Max Size per Dimension: 2022-11-23T01:35:46.1840744Z x 1024(0x400) 2022-11-23T01:35:46.1841066Z y 1024(0x400) 2022-11-23T01:35:46.1841383Z z 1024(0x400) 2022-11-23T01:35:46.1841693Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:35:46.1842021Z Grid Max Size per Dimension: 2022-11-23T01:35:46.1842352Z x 4294967295(0xffffffff) 2022-11-23T01:35:46.1842690Z y 4294967295(0xffffffff) 2022-11-23T01:35:46.1843024Z z 4294967295(0xffffffff) 2022-11-23T01:35:46.1843358Z FBarrier Max Size: 32 2022-11-23T01:35:46.1843632Z *** Done *** 2022-11-23T01:35:46.1862686Z + rocminfo 2022-11-23T01:35:46.1863165Z + grep -E 'Name:.*\sgfx|Marketing' 2022-11-23T01:35:46.2443645Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:35:46.2444722Z Marketing Name: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz 2022-11-23T01:35:46.2445607Z Name: gfx906 2022-11-23T01:35:46.2446397Z Marketing Name: 2022-11-23T01:35:46.2447195Z Name: gfx906 2022-11-23T01:35:46.2447950Z Marketing Name: 2022-11-23T01:35:46.2584526Z + [[ linux-focal-rocm5.2-py3.8 != *-bazel-* ]] 2022-11-23T01:35:46.2585637Z + pip_install --user ninja==1.10.2 2022-11-23T01:35:46.2586743Z + pip install --progress-bar off --user ninja==1.10.2 2022-11-23T01:35:46.7893016Z Collecting ninja==1.10.2 2022-11-23T01:35:46.8363816Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2022-11-23T01:35:47.4787785Z Installing collected packages: ninja 2022-11-23T01:35:47.4869073Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-11-23T01:35:47.4871409Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-11-23T01:35:47.4902349Z Successfully installed ninja-1.10.2 2022-11-23T01:35:47.5506444Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:35:47.5508585Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:35:47.5510792Z + [[ linux-focal-rocm5.2-py3.8 == *asan* ]] 2022-11-23T01:35:47.5511867Z + [[ linux-focal-rocm5.2-py3.8 == *-tsan* ]] 2022-11-23T01:35:47.5512754Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-11-23T01:35:47.5513597Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2022-11-23T01:35:47.5531660Z + [[ linux-focal-rocm5.2-py3.8 == *tbb* ]] 2022-11-23T01:35:47.5572045Z + [[ linux-focal-rocm5.2-py3.8 == *libtorch* ]] 2022-11-23T01:35:47.5573226Z + [[ linux-focal-rocm5.2-py3.8 == *-bazel-* ]] 2022-11-23T01:35:47.5574684Z + [[ linux-focal-rocm5.2-py3.8 == *-tsan* ]] 2022-11-23T01:35:47.5577130Z + cd test 2022-11-23T01:35:47.5578151Z + python -c 'import torch; print(torch.__config__.show())' 2022-11-23T01:35:48.9028694Z PyTorch built with: 2022-11-23T01:35:48.9030424Z - GCC 9.4 2022-11-23T01:35:48.9031746Z - C++ Version: 201402 2022-11-23T01:35:48.9033221Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:35:48.9034761Z - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:35:48.9035837Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:35:48.9036848Z - LAPACK is enabled (usually provided by MKL) 2022-11-23T01:35:48.9037739Z - NNPACK is enabled 2022-11-23T01:35:48.9038573Z - CPU capability usage: AVX2 2022-11-23T01:35:48.9039383Z - HIP Runtime 5.2.21151 2022-11-23T01:35:48.9040375Z - MIOpen 2.17.0 2022-11-23T01:35:48.9041103Z - Magma 2.6.1 2022-11-23T01:35:48.9049992Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, 2022-11-23T01:35:48.9056414Z 2022-11-23T01:35:50.1045996Z + cd test 2022-11-23T01:35:50.1047516Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2022-11-23T01:35:51.3849173Z ATen/Parallel: 2022-11-23T01:35:51.3888332Z at::get_num_threads() : 28 2022-11-23T01:35:51.3889212Z at::get_num_interop_threads() : 28 2022-11-23T01:35:51.3889971Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:35:51.3890681Z omp_get_max_threads() : 28 2022-11-23T01:35:51.3892381Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:35:51.3893892Z mkl_get_max_threads() : 28 2022-11-23T01:35:51.3895054Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:35:51.3895980Z std::thread::hardware_concurrency() : 56 2022-11-23T01:35:51.3896741Z Environment variables: 2022-11-23T01:35:51.3897436Z OMP_NUM_THREADS : [not set] 2022-11-23T01:35:51.3898117Z MKL_NUM_THREADS : [not set] 2022-11-23T01:35:51.3898791Z ATen parallel backend: OpenMP 2022-11-23T01:35:51.3899241Z 2022-11-23T01:35:52.5777785Z + [[ distributed == *backward* ]] 2022-11-23T01:35:52.5779119Z + [[ distributed == *xla* ]] 2022-11-23T01:35:52.5780629Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-11-23T01:35:52.5782039Z + [[ linux-focal-rocm5.2-py3.8 == *libtorch* ]] 2022-11-23T01:35:52.5782875Z + [[ distributed == distributed ]] 2022-11-23T01:35:52.5783557Z + install_filelock 2022-11-23T01:35:52.5784210Z + pip_install filelock 2022-11-23T01:35:52.5785157Z + pip install --progress-bar off filelock 2022-11-23T01:35:53.0528970Z Collecting filelock 2022-11-23T01:35:53.0899678Z Downloading filelock-3.8.0-py3-none-any.whl (10 kB) 2022-11-23T01:35:53.7409081Z Installing collected packages: filelock 2022-11-23T01:35:53.7658155Z Successfully installed filelock-3.8.0 2022-11-23T01:35:53.8250821Z + install_triton 2022-11-23T01:35:53.8251614Z + local commit 2022-11-23T01:35:53.8252238Z + [[ distributed == *rocm* ]] 2022-11-23T01:35:53.8255866Z ++ get_pinned_commit triton 2022-11-23T01:35:53.8256627Z ++ cat .github/ci_commit_pins/triton.txt 2022-11-23T01:35:53.8290743Z + commit=0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:35:53.8292772Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:35:53.8294750Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:35:54.2476961Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:35:54.2479592Z Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-w_ne16m0 2022-11-23T01:35:54.2540682Z Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-w_ne16m0 2022-11-23T01:35:56.3005927Z Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331' 2022-11-23T01:35:56.3074026Z Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:35:57.0305897Z Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:35:57.6224743Z Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:35:57.6229763Z Running command git submodule update --init --recursive -q 2022-11-23T01:35:58.7881065Z Preparing metadata (setup.py) ... [?25l- done 2022-11-23T01:35:58.9718712Z [?25hCollecting cmake 2022-11-23T01:35:59.0143756Z Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB) 2022-11-23T01:36:00.1055611Z Requirement already satisfied: filelock in /opt/conda/lib/python3.8/site-packages (from triton==2.0.0) (3.8.0) 2022-11-23T01:36:00.1058169Z Requirement already satisfied: torch in /opt/conda/lib/python3.8/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385) 2022-11-23T01:36:00.1332311Z Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (2.6.3) 2022-11-23T01:36:00.1334360Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (4.4.0) 2022-11-23T01:36:00.1337399Z Requirement already satisfied: sympy in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (1.11.1) 2022-11-23T01:36:00.1533195Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy->torch->triton==2.0.0) (1.2.1) 2022-11-23T01:36:00.1614688Z Building wheels for collected packages: triton 2022-11-23T01:37:05.3516662Z Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / done 2022-11-23T01:37:05.4110330Z [?25h Created wheel for triton: filename=triton-2.0.0-cp38-cp38-linux_x86_64.whl size=15414539 sha256=abdc9ec7f6b9a5521f03494183222cd84341bb076cf96037d81ca5650460f7ec 2022-11-23T01:37:05.4112018Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/c0/c0/56/bdb2859a55c7764d4e97889d26a8a05b683ef97fe9b1aa7dec 2022-11-23T01:37:05.4152516Z Successfully built triton 2022-11-23T01:37:06.1024114Z Installing collected packages: cmake, triton 2022-11-23T01:37:07.1216557Z Successfully installed cmake-3.25.0 triton-2.0.0 2022-11-23T01:37:07.2238613Z + pip_install --user jinja2 2022-11-23T01:37:07.2239778Z + pip install --progress-bar off --user jinja2 2022-11-23T01:37:07.6964445Z Collecting jinja2 2022-11-23T01:37:07.7580795Z Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) 2022-11-23T01:37:07.7834361Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.8/site-packages (from jinja2) (2.1.1) 2022-11-23T01:37:08.3870872Z Installing collected packages: jinja2 2022-11-23T01:37:08.4601958Z Successfully installed jinja2-3.1.2 2022-11-23T01:37:08.5228642Z + test_distributed 2022-11-23T01:37:08.5229792Z + echo 'Testing distributed python tests' 2022-11-23T01:37:08.5230614Z Testing distributed python tests 2022-11-23T01:37:08.5231813Z + python test/run_test.py --distributed-tests --shard 2 2 --verbose 2022-11-23T01:37:10.8343190Z Ignoring disabled issues: [] 2022-11-23T01:37:10.8499085Z Excluding distributed/rpc/test_faulty_agent on ROCm 2022-11-23T01:37:10.8500123Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm 2022-11-23T01:37:10.8501082Z Excluding distributed/rpc/test_share_memory on ROCm 2022-11-23T01:37:10.8502066Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm 2022-11-23T01:37:10.8503977Z Excluding distributed/_shard/sharding_plan/test_sharding_plan on ROCm 2022-11-23T01:37:10.8505093Z Excluding distributed/_shard/sharded_tensor/test_megatron_prototype on ROCm 2022-11-23T01:37:10.8506196Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor on ROCm 2022-11-23T01:37:10.8507331Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor_reshard on ROCm 2022-11-23T01:37:10.8508426Z Excluding distributed/_shard/sharded_tensor/ops/test_chunk on ROCm 2022-11-23T01:37:10.8509544Z Excluding distributed/_shard/sharded_tensor/ops/test_elementwise_ops on ROCm 2022-11-23T01:37:10.8510648Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding on ROCm 2022-11-23T01:37:10.8511746Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding_bag on ROCm 2022-11-23T01:37:10.8512847Z Excluding distributed/_shard/sharded_tensor/ops/test_binary_cmp on ROCm 2022-11-23T01:37:10.8513882Z Excluding distributed/_shard/sharded_tensor/ops/test_init on ROCm 2022-11-23T01:37:10.8514927Z Excluding distributed/_shard/sharded_tensor/ops/test_linear on ROCm 2022-11-23T01:37:10.8515987Z Excluding distributed/_shard/sharded_tensor/ops/test_math_ops on ROCm 2022-11-23T01:37:10.8517052Z Excluding distributed/_shard/sharded_tensor/ops/test_matrix_ops on ROCm 2022-11-23T01:37:10.8518126Z Excluding distributed/_shard/sharded_tensor/ops/test_softmax on ROCm 2022-11-23T01:37:10.8519188Z Excluding distributed/_shard/sharded_optim/test_sharded_optim on ROCm 2022-11-23T01:37:10.8520607Z Excluding distributed/_shard/test_partial_tensor on ROCm 2022-11-23T01:37:10.8521583Z Excluding distributed/_shard/test_replicated_tensor on ROCm 2022-11-23T01:37:10.8581942Z ##[warning] Gathered no stats from artifacts. Proceeding with default sharding plan. 2022-11-23T01:37:10.8583982Z Selected tests: 2022-11-23T01:37:10.8584810Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:10.8585687Z distributed/test_distributed_spawn 2022-11-23T01:37:10.8586896Z distributed/test_store 2022-11-23T01:37:10.8587528Z distributed/test_nccl 2022-11-23T01:37:10.8588249Z distributed/test_launcher 2022-11-23T01:37:10.8589173Z distributed/test_data_parallel 2022-11-23T01:37:10.8590604Z distributed/test_c10d_spawn_nccl 2022-11-23T01:37:10.8591727Z distributed/test_c10d_pypg 2022-11-23T01:37:10.8592432Z distributed/test_c10d_nccl 2022-11-23T01:37:10.8593128Z distributed/test_c10d_error_logger 2022-11-23T01:37:10.8593993Z distributed/pipeline/sync/test_worker 2022-11-23T01:37:10.8594786Z distributed/pipeline/sync/test_stream 2022-11-23T01:37:10.8595583Z distributed/pipeline/sync/test_pipe 2022-11-23T01:37:10.8596403Z distributed/pipeline/sync/test_microbatch 2022-11-23T01:37:10.8597242Z distributed/pipeline/sync/test_dependency 2022-11-23T01:37:10.8598018Z distributed/pipeline/sync/test_copy 2022-11-23T01:37:10.8598790Z distributed/pipeline/sync/test_bugs 2022-11-23T01:37:10.8599672Z distributed/pipeline/sync/skip/test_verify_skippables 2022-11-23T01:37:10.8600842Z distributed/pipeline/sync/skip/test_stash_pop 2022-11-23T01:37:10.8601686Z distributed/pipeline/sync/skip/test_leak 2022-11-23T01:37:10.8602525Z distributed/pipeline/sync/skip/test_gpipe 2022-11-23T01:37:10.8603369Z distributed/optim/test_zero_redundancy_optimizer 2022-11-23T01:37:10.8604195Z distributed/nn/jit/test_instantiator 2022-11-23T01:37:10.8605144Z distributed/fsdp/test_utils 2022-11-23T01:37:10.8605935Z distributed/fsdp/test_fsdp_use_orig_params 2022-11-23T01:37:10.8606735Z distributed/fsdp/test_fsdp_traversal 2022-11-23T01:37:10.8607556Z distributed/fsdp/test_fsdp_summon_full_params 2022-11-23T01:37:10.8608406Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-11-23T01:37:10.8609214Z distributed/fsdp/test_fsdp_overlap 2022-11-23T01:37:10.8610041Z distributed/fsdp/test_fsdp_multiple_wrapping 2022-11-23T01:37:10.8610894Z distributed/fsdp/test_fsdp_mixed_precision 2022-11-23T01:37:10.8611680Z distributed/fsdp/test_fsdp_meta 2022-11-23T01:37:10.8612428Z distributed/fsdp/test_fsdp_input 2022-11-23T01:37:10.8613179Z distributed/fsdp/test_fsdp_grad_acc 2022-11-23T01:37:10.8614002Z distributed/fsdp/test_fsdp_freezing_weights 2022-11-23T01:37:10.8614823Z distributed/fsdp/test_fsdp_exec_order 2022-11-23T01:37:10.8615606Z distributed/fsdp/test_fsdp_comm_hooks 2022-11-23T01:37:10.8616411Z distributed/fsdp/test_fsdp_clip_grad_norm 2022-11-23T01:37:10.8617177Z distributed/fsdp/test_fsdp_apply 2022-11-23T01:37:10.8617965Z distributed/fsdp/test_checkpoint_wrapper 2022-11-23T01:37:10.8618777Z distributed/elastic/utils/logging_test 2022-11-23T01:37:10.8619596Z distributed/elastic/timer/local_timer_test 2022-11-23T01:37:10.8620381Z distributed/elastic/timer/api_test 2022-11-23T01:37:10.8621150Z distributed/elastic/metrics/api_test 2022-11-23T01:37:10.8621906Z distributed/checkpoint/test_utils 2022-11-23T01:37:10.8622680Z distributed/checkpoint/test_planner 2022-11-23T01:37:10.8623547Z distributed/checkpoint/test_file_system_checkpoint 2022-11-23T01:37:10.8624406Z distributed/checkpoint/test_checkpoint 2022-11-23T01:37:10.8625296Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2022-11-23T01:37:10.8626160Z distributed/_tensor/test_tp_sharding_ops 2022-11-23T01:37:10.8626932Z distributed/_tensor/test_redistribute 2022-11-23T01:37:10.8627706Z distributed/_tensor/test_matrix_ops 2022-11-23T01:37:10.8628464Z distributed/_tensor/test_dtensor 2022-11-23T01:37:10.8629218Z distributed/_tensor/test_common_rules 2022-11-23T01:37:10.8630108Z distributed/_tensor/parallel/test_view_sharding_dim_change 2022-11-23T01:37:10.8631037Z distributed/_tensor/parallel/test_tp_examples 2022-11-23T01:37:10.8631883Z distributed/_tensor/parallel/test_2d_parallel 2022-11-23T01:37:10.8632765Z distributed/_shard/sharding_spec/test_sharding_spec 2022-11-23T01:37:10.8633629Z distributed/_composable/test_replicate 2022-11-23T01:37:10.8634430Z distributed/_composable/test_contract 2022-11-23T01:37:10.8635236Z Prioritized test from test file changes. 2022-11-23T01:37:10.8635966Z reordering tests for PR: 2022-11-23T01:37:10.8636759Z prioritized: [] 2022-11-23T01:37:10.8652703Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/test_store', 'distributed/test_nccl', 'distributed/test_launcher', 'distributed/test_data_parallel', 'distributed/test_c10d_spawn_nccl', 'distributed/test_c10d_pypg', 'distributed/test_c10d_nccl', 'distributed/test_c10d_error_logger', 'distributed/pipeline/sync/test_worker', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/nn/jit/test_instantiator', 'distributed/fsdp/test_utils', 'distributed/fsdp/test_fsdp_use_orig_params', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_input', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/fsdp/test_fsdp_apply', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/elastic/utils/logging_test', 'distributed/elastic/timer/local_timer_test', 'distributed/elastic/timer/api_test', 'distributed/elastic/metrics/api_test', 'distributed/checkpoint/test_utils', 'distributed/checkpoint/test_planner', 'distributed/checkpoint/test_file_system_checkpoint', 'distributed/checkpoint/test_checkpoint', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/_tensor/test_tp_sharding_ops', 'distributed/_tensor/test_redistribute', 'distributed/_tensor/test_matrix_ops', 'distributed/_tensor/test_dtensor', 'distributed/_tensor/test_common_rules', 'distributed/_tensor/parallel/test_view_sharding_dim_change', 'distributed/_tensor/parallel/test_tp_examples', 'distributed/_tensor/parallel/test_2d_parallel', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_composable/test_replicate', 'distributed/_composable/test_contract'] 2022-11-23T01:37:10.8662175Z 2022-11-23T01:37:10.8663680Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2022-11-23T01:37:10.8991470Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2022-11-23T01:37:10.9500482Z parallel (file granularity) tests: 2022-11-23T01:37:10.9501231Z 2022-11-23T01:37:10.9501853Z serial (file granularity) tests: 2022-11-23T01:37:10.9502728Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:10.9503582Z distributed/test_distributed_spawn 2022-11-23T01:37:10.9504310Z distributed/test_store 2022-11-23T01:37:10.9504983Z distributed/test_nccl 2022-11-23T01:37:10.9505650Z distributed/test_launcher 2022-11-23T01:37:10.9506359Z distributed/test_data_parallel 2022-11-23T01:37:10.9507159Z distributed/test_c10d_spawn_nccl 2022-11-23T01:37:10.9507851Z distributed/test_c10d_pypg 2022-11-23T01:37:10.9508536Z distributed/test_c10d_nccl 2022-11-23T01:37:10.9509256Z distributed/test_c10d_error_logger 2022-11-23T01:37:10.9510036Z distributed/pipeline/sync/test_worker 2022-11-23T01:37:10.9510832Z distributed/pipeline/sync/test_stream 2022-11-23T01:37:10.9511625Z distributed/pipeline/sync/test_pipe 2022-11-23T01:37:10.9512426Z distributed/pipeline/sync/test_microbatch 2022-11-23T01:37:10.9513613Z distributed/pipeline/sync/test_dependency 2022-11-23T01:37:10.9514417Z distributed/pipeline/sync/test_copy 2022-11-23T01:37:10.9515385Z distributed/pipeline/sync/test_bugs 2022-11-23T01:37:10.9516266Z distributed/pipeline/sync/skip/test_verify_skippables 2022-11-23T01:37:10.9517194Z distributed/pipeline/sync/skip/test_stash_pop 2022-11-23T01:37:10.9518025Z distributed/pipeline/sync/skip/test_leak 2022-11-23T01:37:10.9518856Z distributed/pipeline/sync/skip/test_gpipe 2022-11-23T01:37:10.9519722Z distributed/optim/test_zero_redundancy_optimizer 2022-11-23T01:37:10.9520782Z distributed/nn/jit/test_instantiator 2022-11-23T01:37:10.9521530Z distributed/fsdp/test_utils 2022-11-23T01:37:10.9522320Z distributed/fsdp/test_fsdp_use_orig_params 2022-11-23T01:37:10.9523107Z distributed/fsdp/test_fsdp_traversal 2022-11-23T01:37:10.9523944Z distributed/fsdp/test_fsdp_summon_full_params 2022-11-23T01:37:10.9524814Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-11-23T01:37:10.9525639Z distributed/fsdp/test_fsdp_overlap 2022-11-23T01:37:10.9526462Z distributed/fsdp/test_fsdp_multiple_wrapping 2022-11-23T01:37:10.9527291Z distributed/fsdp/test_fsdp_mixed_precision 2022-11-23T01:37:10.9528073Z distributed/fsdp/test_fsdp_meta 2022-11-23T01:37:10.9528813Z distributed/fsdp/test_fsdp_input 2022-11-23T01:37:10.9529565Z distributed/fsdp/test_fsdp_grad_acc 2022-11-23T01:37:10.9530535Z distributed/fsdp/test_fsdp_freezing_weights 2022-11-23T01:37:10.9531356Z distributed/fsdp/test_fsdp_exec_order 2022-11-23T01:37:10.9532111Z distributed/fsdp/test_fsdp_comm_hooks 2022-11-23T01:37:10.9532917Z distributed/fsdp/test_fsdp_clip_grad_norm 2022-11-23T01:37:10.9533697Z distributed/fsdp/test_fsdp_apply 2022-11-23T01:37:10.9534481Z distributed/fsdp/test_checkpoint_wrapper 2022-11-23T01:37:10.9535297Z distributed/elastic/utils/logging_test 2022-11-23T01:37:10.9536115Z distributed/elastic/timer/local_timer_test 2022-11-23T01:37:10.9536887Z distributed/elastic/timer/api_test 2022-11-23T01:37:10.9537670Z distributed/elastic/metrics/api_test 2022-11-23T01:37:10.9538445Z distributed/checkpoint/test_utils 2022-11-23T01:37:10.9539221Z distributed/checkpoint/test_planner 2022-11-23T01:37:10.9540086Z distributed/checkpoint/test_file_system_checkpoint 2022-11-23T01:37:10.9540923Z distributed/checkpoint/test_checkpoint 2022-11-23T01:37:10.9541814Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2022-11-23T01:37:10.9542683Z distributed/_tensor/test_tp_sharding_ops 2022-11-23T01:37:10.9543483Z distributed/_tensor/test_redistribute 2022-11-23T01:37:10.9544261Z distributed/_tensor/test_matrix_ops 2022-11-23T01:37:10.9545024Z distributed/_tensor/test_dtensor 2022-11-23T01:37:10.9545753Z distributed/_tensor/test_common_rules 2022-11-23T01:37:10.9546649Z distributed/_tensor/parallel/test_view_sharding_dim_change 2022-11-23T01:37:10.9547572Z distributed/_tensor/parallel/test_tp_examples 2022-11-23T01:37:10.9548437Z distributed/_tensor/parallel/test_2d_parallel 2022-11-23T01:37:10.9549338Z distributed/_shard/sharding_spec/test_sharding_spec 2022-11-23T01:37:10.9550216Z distributed/_composable/test_replicate 2022-11-23T01:37:10.9550993Z distributed/_composable/test_contract 2022-11-23T01:37:13.0319136Z Ignoring disabled issues: [] 2022-11-23T01:37:13.1437903Z Ignoring disabled issues: [] 2022-11-23T01:37:14.4307366Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:37:14.429771] 2022-11-23T01:37:14.4488938Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:37:14.4490691Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2} 2022-11-23T01:37:14.4491768Z Shard 2: test should be run in 1 2022-11-23T01:37:14.4493413Z Running distributed tests for the nccl backend with env init_method in shard 2 of 2 2022-11-23T01:37:14.4501516Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:37:14.449765] 2022-11-23T01:37:49.2250763Z 2022-11-23T01:37:49.2252058Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:49.2268015Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_ne22moe1) 2022-11-23T01:37:49.2271501Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-11-23T01:37:49.2274221Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:49.2275570Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:49.2277239Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:49.2278693Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:49.2280515Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:49.2282969Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:49.2284143Z 2022-11-23T01:37:49.2286048Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:49.2287972Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2289191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2290801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2292041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2292635Z 2022-11-23T01:37:49.2292877Z Running tests... 2022-11-23T01:37:49.2293983Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2295274Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s) 2022-11-23T01:37:49.2296206Z 2022-11-23T01:37:49.2296908Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2297760Z Ran 1 test in 0.001s 2022-11-23T01:37:49.2298158Z 2022-11-23T01:37:49.2298401Z OK (skipped=1) 2022-11-23T01:37:49.2298774Z 2022-11-23T01:37:49.2299063Z Generating XML reports... 2022-11-23T01:37:49.2300918Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013717.xml 2022-11-23T01:37:49.2303099Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:49.2304977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2306164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2307738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2308981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2309568Z 2022-11-23T01:37:49.2309810Z Running tests... 2022-11-23T01:37:49.2310919Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2312193Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s) 2022-11-23T01:37:49.2312942Z 2022-11-23T01:37:49.2313661Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2314499Z Ran 1 test in 0.001s 2022-11-23T01:37:49.2315103Z 2022-11-23T01:37:49.2315345Z OK (skipped=1) 2022-11-23T01:37:49.2315719Z 2022-11-23T01:37:49.2316007Z Generating XML reports... 2022-11-23T01:37:49.2317858Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013720.xml 2022-11-23T01:37:49.2320028Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:49.2322135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2323324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2324901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2326128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2326707Z 2022-11-23T01:37:49.2326951Z Running tests... 2022-11-23T01:37:49.2328062Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2329440Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1240 2022-11-23T01:37:49.2330980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1241 2022-11-23T01:37:49.2332334Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:49.2334147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2335345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2336910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2338154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2339332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:49.2341030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2342217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2343806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2345055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2346192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:49.2347185Z skip: Test skipped for ROCm (3.517s) 2022-11-23T01:37:49.2347647Z 2022-11-23T01:37:49.2348390Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2349231Z Ran 1 test in 3.518s 2022-11-23T01:37:49.2349629Z 2022-11-23T01:37:49.2349880Z OK (skipped=1) 2022-11-23T01:37:49.2350253Z 2022-11-23T01:37:49.2350540Z Generating XML reports... 2022-11-23T01:37:49.2352387Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013722.xml 2022-11-23T01:37:49.2354535Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:49.2356394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2357573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2359145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2360551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2361142Z 2022-11-23T01:37:49.2361387Z Running tests... 2022-11-23T01:37:49.2362347Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2362962Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1417 2022-11-23T01:37:49.2363511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1418 2022-11-23T01:37:49.2364042Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:49.2364748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2365221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2365844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2366331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2366791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:49.2367447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2367911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2368588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2369081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2369539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:49.2369941Z skip: Test skipped for ROCm (3.506s) 2022-11-23T01:37:49.2370124Z 2022-11-23T01:37:49.2370414Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2370738Z Ran 1 test in 3.507s 2022-11-23T01:37:49.2370897Z 2022-11-23T01:37:49.2370998Z OK (skipped=1) 2022-11-23T01:37:49.2371156Z 2022-11-23T01:37:49.2371276Z Generating XML reports... 2022-11-23T01:37:49.2372000Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013729.xml 2022-11-23T01:37:49.2372845Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:49.2373578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2374055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2374657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2375150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2375383Z 2022-11-23T01:37:49.2375483Z Running tests... 2022-11-23T01:37:49.2375924Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2376486Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1594 2022-11-23T01:37:49.2377052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1595 2022-11-23T01:37:49.2377587Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:49.2378294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2378751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2379369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2379854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2380314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:49.2380889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:37:49.2381575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2382046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2382655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2383140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2383601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:49.2384117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:37:49.2384835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:49.2385409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:37:49.2386114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:49.2386734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:37:49.2387433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:49.2388175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:49.2388583Z ok (4.007s) 2022-11-23T01:37:49.2388727Z 2022-11-23T01:37:49.2389016Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2389357Z Ran 1 test in 4.008s 2022-11-23T01:37:49.2389517Z 2022-11-23T01:37:49.2389604Z OK 2022-11-23T01:37:49.2389737Z 2022-11-23T01:37:49.2389855Z Generating XML reports... 2022-11-23T01:37:49.2390569Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013735.xml 2022-11-23T01:37:49.2391420Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:49.2392145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2392619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2393239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2393731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2393962Z 2022-11-23T01:37:49.2394063Z Running tests... 2022-11-23T01:37:49.2394481Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2395039Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1786 2022-11-23T01:37:49.2395594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1787 2022-11-23T01:37:49.2396119Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:49.2396825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2397294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2397919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2398407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2398851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:49.2399426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:37:49.2400118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:49.2400670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:49.2401298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:49.2401788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:49.2402244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:49.2402744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:37:49.2403458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:49.2404025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:37:49.2404725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:49.2405345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:37:49.2406057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:49.2406803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:49.2407206Z ok (4.008s) 2022-11-23T01:37:49.2407337Z 2022-11-23T01:37:49.2407629Z ---------------------------------------------------------------------- 2022-11-23T01:37:49.2407973Z Ran 1 test in 4.008s 2022-11-23T01:37:49.2408131Z 2022-11-23T01:37:49.2408220Z OK 2022-11-23T01:37:49.2408353Z 2022-11-23T01:37:49.2408475Z Generating XML reports... 2022-11-23T01:37:49.2409202Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013742.xml 2022-11-23T01:37:49.2409630Z 2022-11-23T01:37:49.2410010Z ##[endgroup] 2022-11-23T01:37:49.2410783Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_ne22moe1) 2022-11-23T01:37:49.2411242Z 2022-11-23T01:37:49.2411457Z Running distributed tests for the nccl backend with file init_method in shard 2 of 2 2022-11-23T01:37:49.2412389Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:37:49.228488] 2022-11-23T01:38:23.7683388Z 2022-11-23T01:38:23.7684572Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:38:23.7691969Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_klmvueuk) 2022-11-23T01:38:23.7695225Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-11-23T01:38:23.7697586Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:23.7698557Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:23.7699540Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:23.7700937Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:23.7701892Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:23.7702887Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:23.7703790Z 2022-11-23T01:38:23.7706124Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:23.7708083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7709279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7710884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7712120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7712726Z 2022-11-23T01:38:23.7712969Z Running tests... 2022-11-23T01:38:23.7714093Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7715366Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s) 2022-11-23T01:38:23.7716255Z 2022-11-23T01:38:23.7717343Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7718215Z Ran 1 test in 0.001s 2022-11-23T01:38:23.7718611Z 2022-11-23T01:38:23.7718848Z OK (skipped=1) 2022-11-23T01:38:23.7719220Z 2022-11-23T01:38:23.7719488Z Generating XML reports... 2022-11-23T01:38:23.7721528Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013752.xml 2022-11-23T01:38:23.7724533Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:23.7727768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7729692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7732364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7733629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7734222Z 2022-11-23T01:38:23.7734466Z Running tests... 2022-11-23T01:38:23.7735573Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7736852Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s) 2022-11-23T01:38:23.7737608Z 2022-11-23T01:38:23.7738307Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7739129Z Ran 1 test in 0.001s 2022-11-23T01:38:23.7739525Z 2022-11-23T01:38:23.7739767Z OK (skipped=1) 2022-11-23T01:38:23.7740141Z 2022-11-23T01:38:23.7740436Z Generating XML reports... 2022-11-23T01:38:23.7742276Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013754.xml 2022-11-23T01:38:23.7744441Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:23.7746308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7747484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7749070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7750324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7750913Z 2022-11-23T01:38:23.7751155Z Running tests... 2022-11-23T01:38:23.7752249Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7753868Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2152 2022-11-23T01:38:23.7755271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2153 2022-11-23T01:38:23.7756624Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:23.7758401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7759580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7761524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7762762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7763929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:23.7765636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7766819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7768528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7769767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7770927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:23.7771923Z skip: Test skipped for ROCm (3.305s) 2022-11-23T01:38:23.7772382Z 2022-11-23T01:38:23.7773130Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7773974Z Ran 1 test in 3.306s 2022-11-23T01:38:23.7774370Z 2022-11-23T01:38:23.7774610Z OK (skipped=1) 2022-11-23T01:38:23.7774986Z 2022-11-23T01:38:23.7775255Z Generating XML reports... 2022-11-23T01:38:23.7777103Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013757.xml 2022-11-23T01:38:23.7779261Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:23.7781136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7782324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7783904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7785149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7785736Z 2022-11-23T01:38:23.7785982Z Running tests... 2022-11-23T01:38:23.7787063Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7788450Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2329 2022-11-23T01:38:23.7789847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2330 2022-11-23T01:38:23.7791185Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:23.7792976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7794165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7795737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7796953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7798110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:23.7799809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7801303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7802889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7804110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7805257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:23.7805963Z skip: Test skipped for ROCm (3.305s) 2022-11-23T01:38:23.7806146Z 2022-11-23T01:38:23.7806434Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7806770Z Ran 1 test in 3.306s 2022-11-23T01:38:23.7806927Z 2022-11-23T01:38:23.7807026Z OK (skipped=1) 2022-11-23T01:38:23.7807174Z 2022-11-23T01:38:23.7807294Z Generating XML reports... 2022-11-23T01:38:23.7807997Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013803.xml 2022-11-23T01:38:23.7808833Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:23.7809618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7810160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7836471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7837092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7837334Z 2022-11-23T01:38:23.7837430Z Running tests... 2022-11-23T01:38:23.7837936Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7838494Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2506 2022-11-23T01:38:23.7839071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2507 2022-11-23T01:38:23.7839605Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:23.7840413Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7840892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7841522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7842009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7842462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:23.7842969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:38:23.7843647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7844108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7844738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7845239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7845696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:23.7846211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:38:23.7846926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:23.7847470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:38:23.7848337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:23.7848894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:38:23.7849591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:38:23.7850328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:38:23.7850727Z ok (4.016s) 2022-11-23T01:38:23.7850867Z 2022-11-23T01:38:23.7851156Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7851486Z Ran 1 test in 4.016s 2022-11-23T01:38:23.7851642Z 2022-11-23T01:38:23.7851725Z OK 2022-11-23T01:38:23.7851852Z 2022-11-23T01:38:23.7851957Z Generating XML reports... 2022-11-23T01:38:23.7852689Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013810.xml 2022-11-23T01:38:23.7853526Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:23.7854310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7854776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7855396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7855879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7856110Z 2022-11-23T01:38:23.7856212Z Running tests... 2022-11-23T01:38:23.7856635Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7857188Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2698 2022-11-23T01:38:23.7857898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2699 2022-11-23T01:38:23.7858420Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:23.7859118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7859584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7860206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7860680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7861131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:23.7861640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:38:23.7862321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:23.7862789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:23.7863406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:23.7863890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:23.7864339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:23.7864841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:38:23.7865547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:23.7866109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:38:23.7866907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:23.7867458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:38:23.7868157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:38:23.7868900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:38:23.7869305Z ok (4.008s) 2022-11-23T01:38:23.7869435Z 2022-11-23T01:38:23.7869728Z ---------------------------------------------------------------------- 2022-11-23T01:38:23.7870071Z Ran 1 test in 4.008s 2022-11-23T01:38:23.7870232Z 2022-11-23T01:38:23.7870314Z OK 2022-11-23T01:38:23.7870442Z 2022-11-23T01:38:23.7870562Z Generating XML reports... 2022-11-23T01:38:23.7871296Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013816.xml 2022-11-23T01:38:23.7871721Z 2022-11-23T01:38:23.7872115Z ##[endgroup] 2022-11-23T01:38:23.7872940Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_klmvueuk) 2022-11-23T01:38:23.7873400Z 2022-11-23T01:38:23.7873524Z Shard 2: gloo should be run in 1 2022-11-23T01:38:23.7873991Z Running distributed/test_distributed_spawn ... [2022-11-23 01:38:23.771363] 2022-11-23T01:38:23.7902404Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:38:23.7903380Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 1} 2022-11-23T01:38:23.7903812Z Shard 2: test should be run in 1 2022-11-23T01:38:23.7904197Z Running distributed tests for the nccl backend with env init_method in shard 2 of 2 2022-11-23T01:38:23.7909957Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:38:23.790540] 2022-11-23T02:00:45.4864514Z 2022-11-23T02:00:45.4865547Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T02:00:45.4869129Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_7ddbvpx2) 2022-11-23T02:00:45.4880832Z 2022-11-23T02:00:45.4986533Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-11-23T02:00:45.5035672Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5039542Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5040005Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5040491Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5040986Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5041489Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5042019Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5042554Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5043089Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5043675Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5044270Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5044840Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5045405Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5045966Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5046475Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5046976Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5047474Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5047933Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5048402Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5048874Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5049392Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5049896Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5050420Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5050837Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5051294Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5051743Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5052187Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5052647Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5053076Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5053466Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5053885Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5054303Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5054724Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5055152Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5055596Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5056021Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5056514Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5056954Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5057395Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5057818Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5058265Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5058711Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5059178Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5059662Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5060120Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5060578Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5061031Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5061497Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5061909Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5062383Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5062845Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5063284Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5063715Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5064167Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5064616Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5065028Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5065479Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5065910Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5066344Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5066751Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5067186Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5067589Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5067985Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5068387Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5068864Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5069295Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5069727Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5070138Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5070527Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5070930Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5071340Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5071754Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5072176Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5072608Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5072997Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5073405Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5073799Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5074210Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5074630Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5075099Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5075505Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5075913Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5076345Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5076794Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5077249Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5077730Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5078208Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5078668Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5079140Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5079603Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5080064Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5080593Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5081056Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5081525Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5081999Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5082485Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5082970Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5083441Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5083894Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5084301Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5084696Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5085062Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5085447Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5085848Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5086271Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5086679Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5087138Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5087540Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5087969Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5088387Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5088809Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5089249Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5089699Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5090125Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5090559Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5090999Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5091438Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5091893Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5092348Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5092783Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5093225Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5093631Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5094041Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5094456Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5094870Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5095296Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5095761Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5096298Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5096783Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5097217Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5097667Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5098123Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5098592Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5099079Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5099541Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5099993Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5100457Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5100890Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5101288Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5101693Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5102123Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5102560Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5103026Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5103493Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5103932Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5104373Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5104847Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5105427Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5106031Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5106683Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5107333Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5107985Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5108620Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5109265Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5109900Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5110577Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5111131Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5111649Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5112120Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5112524Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5112950Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5113377Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5113800Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5114254Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5114709Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5115195Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5115697Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5116131Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5116555Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5116971Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5117427Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5117884Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5118321Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5118765Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5119210Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5119671Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5120098Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5120575Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5121011Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5121451Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5121883Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5122325Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5122869Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5123311Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5123730Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5124138Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5124556Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5124995Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5125435Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5125814Z test_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5126203Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5126590Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5126983Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5127385Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5127772Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5128161Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5128567Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5128997Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5129381Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5129772Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5130183Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5130601Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5130973Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5131332Z test_isend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5131738Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5132164Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5132596Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5133080Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5133540Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5133976Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5134422Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5134879Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5135339Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5135784Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5136221Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5136657Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5137086Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5137522Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5137930Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5138359Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5138846Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5139507Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5140008Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5140480Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5140957Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5141493Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5141935Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5142372Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5142813Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5143273Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5143749Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5144257Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5144800Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5145310Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5145761Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5146157Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5146577Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5147005Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5147448Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5147858Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5148265Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5148678Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5149047Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5149423Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5149820Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5150215Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5150643Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5151072Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5151447Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5151829Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5152245Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5152646Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5153021Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5153416Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5153795Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5154183Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5154594Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5155002Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5155398Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5155803Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5156176Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5156580Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5157020Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5157492Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5157935Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5158344Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5158773Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5159204Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5159640Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5160109Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5160607Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5161064Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5161512Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5161920Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5162349Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5162763Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5163163Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5163610Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5164084Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5164561Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5165472Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5166248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5166739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5167398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5167896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5168132Z 2022-11-23T02:00:45.5168234Z Running tests... 2022-11-23T02:00:45.5168683Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5169321Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2949 2022-11-23T02:00:45.5169977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2950 2022-11-23T02:00:45.5170510Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5171223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5171700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5172324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5172818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5173300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5173971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5174443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5175065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5175554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5176030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5176748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5177497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5178049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5178595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5179147Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:00:45.5180069Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:00:45.5180779Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:00:45.5181678Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:00:45.5182384Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:00:45.5183288Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:00:45.5183992Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:00:45.5184925Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:00:45.5185633Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:00:45.5186529Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:00:45.5187225Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:00:45.5188112Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:00:45.5188814Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:00:45.5189690Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:00:45.5190384Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:00:45.5191272Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:00:45.5191767Z ok (4.629s) 2022-11-23T02:00:45.5191912Z 2022-11-23T02:00:45.5192201Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5192540Z Ran 1 test in 4.630s 2022-11-23T02:00:45.5192699Z 2022-11-23T02:00:45.5192782Z OK 2022-11-23T02:00:45.5192909Z 2022-11-23T02:00:45.5193012Z Generating XML reports... 2022-11-23T02:00:45.5193684Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013827.xml 2022-11-23T02:00:45.5194399Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5195069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5195540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5196164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5196656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5196886Z 2022-11-23T02:00:45.5196989Z Running tests... 2022-11-23T02:00:45.5197411Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5197992Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s) 2022-11-23T02:00:45.5198310Z 2022-11-23T02:00:45.5198599Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5198945Z Ran 1 test in 0.003s 2022-11-23T02:00:45.5199106Z 2022-11-23T02:00:45.5199208Z OK (skipped=1) 2022-11-23T02:00:45.5199357Z 2022-11-23T02:00:45.5199479Z Generating XML reports... 2022-11-23T02:00:45.5200182Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013834.xml 2022-11-23T02:00:45.5200883Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5201551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5202023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5202648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5203147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5203380Z 2022-11-23T02:00:45.5203485Z Running tests... 2022-11-23T02:00:45.5204008Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5204561Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3197 2022-11-23T02:00:45.5205135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3198 2022-11-23T02:00:45.5205666Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5206380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5206854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5207481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5207974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5208463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5209128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5209592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5210209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5210695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5211170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5211892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5212645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5213187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5213678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5214022Z ok (3.630s) 2022-11-23T02:00:45.5214166Z 2022-11-23T02:00:45.5214459Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5214800Z Ran 1 test in 3.630s 2022-11-23T02:00:45.5214960Z 2022-11-23T02:00:45.5215042Z OK 2022-11-23T02:00:45.5215171Z 2022-11-23T02:00:45.5215288Z Generating XML reports... 2022-11-23T02:00:45.5215998Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013837.xml 2022-11-23T02:00:45.5216715Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5217390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5217863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5218479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5218968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5219201Z 2022-11-23T02:00:45.5219304Z Running tests... 2022-11-23T02:00:45.5219727Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5221030Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.511s) 2022-11-23T02:00:45.5221718Z 2022-11-23T02:00:45.5222056Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5222385Z Ran 1 test in 0.511s 2022-11-23T02:00:45.5222547Z 2022-11-23T02:00:45.5222648Z OK (skipped=1) 2022-11-23T02:00:45.5222802Z 2022-11-23T02:00:45.5222921Z Generating XML reports... 2022-11-23T02:00:45.5223584Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013844.xml 2022-11-23T02:00:45.5224286Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5224962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5225445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5226058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5226549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5226781Z 2022-11-23T02:00:45.5226884Z Running tests... 2022-11-23T02:00:45.5227318Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5227871Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.001s) 2022-11-23T02:00:45.5228213Z 2022-11-23T02:00:45.5228498Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5228836Z Ran 1 test in 0.001s 2022-11-23T02:00:45.5228994Z 2022-11-23T02:00:45.5229079Z OK (skipped=1) 2022-11-23T02:00:45.5229227Z 2022-11-23T02:00:45.5229343Z Generating XML reports... 2022-11-23T02:00:45.5230000Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013847.xml 2022-11-23T02:00:45.5230703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5231372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5231837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5232455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5232933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5233164Z 2022-11-23T02:00:45.5233262Z Running tests... 2022-11-23T02:00:45.5233692Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5234256Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.001s) 2022-11-23T02:00:45.5234669Z 2022-11-23T02:00:45.5234954Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5235289Z Ran 1 test in 0.001s 2022-11-23T02:00:45.5235446Z 2022-11-23T02:00:45.5235545Z OK (skipped=1) 2022-11-23T02:00:45.5235698Z 2022-11-23T02:00:45.5235801Z Generating XML reports... 2022-11-23T02:00:45.5236452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013850.xml 2022-11-23T02:00:45.5237162Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5237830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5238305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5238935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5239428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5239659Z 2022-11-23T02:00:45.5239759Z Running tests... 2022-11-23T02:00:45.5240236Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5241013Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3550 2022-11-23T02:00:45.5241635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3551 2022-11-23T02:00:45.5242166Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5242881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5243345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5272620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5273142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5273630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5274319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5274776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5275387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5275861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5276330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5277041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5277774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5278314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5278786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5279304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsw94muj9 2022-11-23T02:00:45.5279860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsw94muj9/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5280463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kwi7j35 2022-11-23T02:00:45.5281008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kwi7j35/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5282204Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5282772Z warnings.warn( 2022-11-23T02:00:45.5283151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5284092Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5284647Z warnings.warn( 2022-11-23T02:00:45.5285022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5285508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5285996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5286480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5286959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5287499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5287988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5288469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5288942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5289417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5289901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5290246Z ok (7.749s) 2022-11-23T02:00:45.5290383Z 2022-11-23T02:00:45.5290676Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5291000Z Ran 1 test in 7.750s 2022-11-23T02:00:45.5291152Z 2022-11-23T02:00:45.5291227Z OK 2022-11-23T02:00:45.5291341Z 2022-11-23T02:00:45.5291450Z Generating XML reports... 2022-11-23T02:00:45.5292110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013853.xml 2022-11-23T02:00:45.5292817Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5293476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5293934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5294548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5295034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5295261Z 2022-11-23T02:00:45.5295347Z Running tests... 2022-11-23T02:00:45.5295771Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5296391Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3746 2022-11-23T02:00:45.5296997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3747 2022-11-23T02:00:45.5297512Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5298205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5298666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5299334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5299810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5300277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5301097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5301546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5302153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5302623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5303083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5303777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5304512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5305047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5305572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5306084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pbe3fou 2022-11-23T02:00:45.5306644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pbe3fou/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5307190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9siubhtb 2022-11-23T02:00:45.5307738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9siubhtb/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5308292Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5308853Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5309832Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5310390Z warnings.warn( 2022-11-23T02:00:45.5310765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5311710Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5312267Z warnings.warn( 2022-11-23T02:00:45.5312649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5313140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5313632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5313983Z ok (4.543s) 2022-11-23T02:00:45.5314124Z 2022-11-23T02:00:45.5314415Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5314734Z Ran 1 test in 4.544s 2022-11-23T02:00:45.5314885Z 2022-11-23T02:00:45.5314959Z OK 2022-11-23T02:00:45.5315079Z 2022-11-23T02:00:45.5315194Z Generating XML reports... 2022-11-23T02:00:45.5315841Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013904.xml 2022-11-23T02:00:45.5316552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5317222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5317743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5318363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5318857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5319083Z 2022-11-23T02:00:45.5319177Z Running tests... 2022-11-23T02:00:45.5319596Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5320324Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3940 2022-11-23T02:00:45.5320962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3941 2022-11-23T02:00:45.5321482Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5322191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5322660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5323331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5323820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5324283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5324957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5325415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5326030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5326508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5326976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5327682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5328431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5328968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5329447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5329966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5h9c8jt 2022-11-23T02:00:45.5330533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5h9c8jt/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5331087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevh3iew8 2022-11-23T02:00:45.5331651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevh3iew8/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5332644Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5333209Z warnings.warn( 2022-11-23T02:00:45.5333581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5334528Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5335135Z warnings.warn( 2022-11-23T02:00:45.5335517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5336019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5336513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5337006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5337493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5337982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5338457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5338806Z ok (5.039s) 2022-11-23T02:00:45.5338948Z 2022-11-23T02:00:45.5339243Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5339573Z Ran 1 test in 5.040s 2022-11-23T02:00:45.5339726Z 2022-11-23T02:00:45.5339807Z OK 2022-11-23T02:00:45.5339923Z 2022-11-23T02:00:45.5340036Z Generating XML reports... 2022-11-23T02:00:45.5340688Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013911.xml 2022-11-23T02:00:45.5341445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5342117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5342576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5343194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5343673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5343900Z 2022-11-23T02:00:45.5343986Z Running tests... 2022-11-23T02:00:45.5344415Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5345067Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4134 2022-11-23T02:00:45.5345714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4135 2022-11-23T02:00:45.5346242Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5346933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5347403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5348013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5348488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5348963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5349631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5350095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5350708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5351188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5351656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5352353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5353093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5353685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5354166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5354684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_4bb_4t 2022-11-23T02:00:45.5355244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_4bb_4t/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5355796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_kkj0ou 2022-11-23T02:00:45.5356340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_kkj0ou/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5357312Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5357890Z warnings.warn( 2022-11-23T02:00:45.5358277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5359267Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5359826Z warnings.warn( 2022-11-23T02:00:45.5360268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5360624Z ok (4.843s) 2022-11-23T02:00:45.5360770Z 2022-11-23T02:00:45.5361066Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5361390Z Ran 1 test in 4.844s 2022-11-23T02:00:45.5361547Z 2022-11-23T02:00:45.5361629Z OK 2022-11-23T02:00:45.5361754Z 2022-11-23T02:00:45.5361868Z Generating XML reports... 2022-11-23T02:00:45.5362532Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013919.xml 2022-11-23T02:00:45.5363244Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5363909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5364366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5364986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5365473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5365706Z 2022-11-23T02:00:45.5365805Z Running tests... 2022-11-23T02:00:45.5366228Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5366862Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4328 2022-11-23T02:00:45.5367522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4329 2022-11-23T02:00:45.5368041Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5368723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5369188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5369806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5370286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5370755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5371517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5371973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5372589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5373064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5373532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5374245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5374984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5375530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5376025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5376539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppj9nk8t5 2022-11-23T02:00:45.5377094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppj9nk8t5/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5377699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u9pk325 2022-11-23T02:00:45.5378256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u9pk325/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5379239Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5379797Z warnings.warn( 2022-11-23T02:00:45.5380182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5381138Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5381695Z warnings.warn( 2022-11-23T02:00:45.5382067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5382570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5383059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5383406Z ok (7.249s) 2022-11-23T02:00:45.5383546Z 2022-11-23T02:00:45.5383834Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5384159Z Ran 1 test in 7.249s 2022-11-23T02:00:45.5384311Z 2022-11-23T02:00:45.5384390Z OK 2022-11-23T02:00:45.5384513Z 2022-11-23T02:00:45.5384621Z Generating XML reports... 2022-11-23T02:00:45.5385272Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013927.xml 2022-11-23T02:00:45.5385976Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5386642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5387103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5387714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5388204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5388428Z 2022-11-23T02:00:45.5388514Z Running tests... 2022-11-23T02:00:45.5388941Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5389564Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4524 2022-11-23T02:00:45.5390232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4525 2022-11-23T02:00:45.5390753Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5391450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5391903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5392517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5392987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5393447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5394128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5394584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5395235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5395716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5396188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5396884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5397626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5398172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5398659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5399176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wq7avjp 2022-11-23T02:00:45.5399743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wq7avjp/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5400347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkogztwji 2022-11-23T02:00:45.5401068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkogztwji/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5402046Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5402612Z warnings.warn( 2022-11-23T02:00:45.5402998Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5403955Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5404519Z warnings.warn( 2022-11-23T02:00:45.5404905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5405409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5405899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5406239Z ok (6.461s) 2022-11-23T02:00:45.5406376Z 2022-11-23T02:00:45.5406664Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5406994Z Ran 1 test in 6.461s 2022-11-23T02:00:45.5407149Z 2022-11-23T02:00:45.5407231Z OK 2022-11-23T02:00:45.5407417Z 2022-11-23T02:00:45.5407529Z Generating XML reports... 2022-11-23T02:00:45.5408193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013938.xml 2022-11-23T02:00:45.5408899Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5409561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5410023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5410640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5411117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5411352Z 2022-11-23T02:00:45.5411451Z Running tests... 2022-11-23T02:00:45.5411872Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5412519Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4720 2022-11-23T02:00:45.5413143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4721 2022-11-23T02:00:45.5413704Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5414400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5414871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5415482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5415966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5416433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5417111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5417564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5418177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5418656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5419119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5419819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5420559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5421099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5421581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5422108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxf759ix 2022-11-23T02:00:45.5422672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxf759ix/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5423219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbb8_2sn 2022-11-23T02:00:45.5423772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbb8_2sn/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5424339Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5424898Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5425925Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5426484Z warnings.warn( 2022-11-23T02:00:45.5426859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5427807Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.5428375Z warnings.warn( 2022-11-23T02:00:45.5428752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5429253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5429753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5430104Z ok (4.550s) 2022-11-23T02:00:45.5430233Z 2022-11-23T02:00:45.5430525Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5430862Z Ran 1 test in 4.551s 2022-11-23T02:00:45.5431014Z 2022-11-23T02:00:45.5431093Z OK 2022-11-23T02:00:45.5431218Z 2022-11-23T02:00:45.5431380Z Generating XML reports... 2022-11-23T02:00:45.5432039Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013947.xml 2022-11-23T02:00:45.5432741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5433405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5433861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5434473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5434957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5435180Z 2022-11-23T02:00:45.5435278Z Running tests... 2022-11-23T02:00:45.5435704Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5437003Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.515s) 2022-11-23T02:00:45.5437669Z 2022-11-23T02:00:45.5437949Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5438276Z Ran 1 test in 0.516s 2022-11-23T02:00:45.5438432Z 2022-11-23T02:00:45.5438518Z OK (skipped=1) 2022-11-23T02:00:45.5438670Z 2022-11-23T02:00:45.5438782Z Generating XML reports... 2022-11-23T02:00:45.5439434Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013955.xml 2022-11-23T02:00:45.5440184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5440860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5441324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5441940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5442424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5442640Z 2022-11-23T02:00:45.5442738Z Running tests... 2022-11-23T02:00:45.5443172Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5443838Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4972 2022-11-23T02:00:45.5444438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4973 2022-11-23T02:00:45.5444965Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5445666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5446134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5446741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5447233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5447699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5448376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5448838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5449502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5449986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5450458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5451164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5451902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5452447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5452934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5453282Z ok (3.852s) 2022-11-23T02:00:45.5453421Z 2022-11-23T02:00:45.5453706Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5454039Z Ran 1 test in 3.853s 2022-11-23T02:00:45.5454185Z 2022-11-23T02:00:45.5454265Z OK 2022-11-23T02:00:45.5454388Z 2022-11-23T02:00:45.5454503Z Generating XML reports... 2022-11-23T02:00:45.5455165Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013958.xml 2022-11-23T02:00:45.5455873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5456535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5457002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5457615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5458080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5458306Z 2022-11-23T02:00:45.5458404Z Running tests... 2022-11-23T02:00:45.5458843Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5460144Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.516s) 2022-11-23T02:00:45.5460820Z 2022-11-23T02:00:45.5461102Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5461478Z Ran 1 test in 0.516s 2022-11-23T02:00:45.5461634Z 2022-11-23T02:00:45.5461729Z OK (skipped=1) 2022-11-23T02:00:45.5461875Z 2022-11-23T02:00:45.5461989Z Generating XML reports... 2022-11-23T02:00:45.5462630Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014005.xml 2022-11-23T02:00:45.5463341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5464000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5464472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5465085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5465568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5465798Z 2022-11-23T02:00:45.5465898Z Running tests... 2022-11-23T02:00:45.5466334Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5466902Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5209 2022-11-23T02:00:45.5467522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5210 2022-11-23T02:00:45.5468046Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5468742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5469207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5469814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5470296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5470758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5471423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5471887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5472508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5472984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5473457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5474164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5474906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5475443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5475927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5476262Z ok (4.352s) 2022-11-23T02:00:45.5476402Z 2022-11-23T02:00:45.5476692Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5477023Z Ran 1 test in 4.353s 2022-11-23T02:00:45.5477187Z 2022-11-23T02:00:45.5477270Z OK 2022-11-23T02:00:45.5477396Z 2022-11-23T02:00:45.5477511Z Generating XML reports... 2022-11-23T02:00:45.5478158Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014009.xml 2022-11-23T02:00:45.5478856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5479514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5480023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5480700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5481192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5481423Z 2022-11-23T02:00:45.5481520Z Running tests... 2022-11-23T02:00:45.5481935Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5482426Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-11-23T02:00:45.5482722Z 2022-11-23T02:00:45.5483003Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5483333Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5483488Z 2022-11-23T02:00:45.5483587Z OK (skipped=1) 2022-11-23T02:00:45.5483732Z 2022-11-23T02:00:45.5483851Z Generating XML reports... 2022-11-23T02:00:45.5484504Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014016.xml 2022-11-23T02:00:45.5485202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5485911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5486374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5486989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5487471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5487704Z 2022-11-23T02:00:45.5487802Z Running tests... 2022-11-23T02:00:45.5488225Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5488651Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5489212Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5453 2022-11-23T02:00:45.5489763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5454 2022-11-23T02:00:45.5490285Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5490982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5491450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5492066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5492547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5493006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5493677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5494121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5494738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5495214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5495683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5496393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5497121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5497659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5498202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5498719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmkeqknkj 2022-11-23T02:00:45.5499287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmkeqknkj/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5499835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mfh9ykp 2022-11-23T02:00:45.5500390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mfh9ykp/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5500783Z ok (6.852s) 2022-11-23T02:00:45.5500913Z 2022-11-23T02:00:45.5501204Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5501692Z Ran 1 test in 6.852s 2022-11-23T02:00:45.5501849Z 2022-11-23T02:00:45.5501931Z OK 2022-11-23T02:00:45.5502057Z 2022-11-23T02:00:45.5502178Z Generating XML reports... 2022-11-23T02:00:45.5502836Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014020.xml 2022-11-23T02:00:45.5503542Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5504264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5504733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5505349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5505835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5506064Z 2022-11-23T02:00:45.5506162Z Running tests... 2022-11-23T02:00:45.5506598Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5507068Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5507655Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5649 2022-11-23T02:00:45.5508195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5650 2022-11-23T02:00:45.5508726Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5509427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5509889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5510501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5510981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5511455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5512118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5512584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5513200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5513681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5514143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5514849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5515595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5516189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5516663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5517187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhnv73lz 2022-11-23T02:00:45.5517754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhnv73lz/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5518304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10qne9s0 2022-11-23T02:00:45.5518856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10qne9s0/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5519387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5519888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5520294Z ok (6.969s) 2022-11-23T02:00:45.5520430Z 2022-11-23T02:00:45.5520725Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5521064Z Ran 1 test in 6.969s 2022-11-23T02:00:45.5521224Z 2022-11-23T02:00:45.5521307Z OK 2022-11-23T02:00:45.5521435Z 2022-11-23T02:00:45.5521553Z Generating XML reports... 2022-11-23T02:00:45.5522264Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014029.xml 2022-11-23T02:00:45.5522978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5523632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5524099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5524720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5525211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5525442Z 2022-11-23T02:00:45.5525541Z Running tests... 2022-11-23T02:00:45.5525969Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5526453Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5527079Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5845 2022-11-23T02:00:45.5527643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5846 2022-11-23T02:00:45.5528176Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5528878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5529348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5529974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5530457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5530934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5531598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5532065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5532681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5533166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5533634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5534396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5535139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5535692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5536173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5536700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai4640wr 2022-11-23T02:00:45.5537265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai4640wr/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5537823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52d77qpx 2022-11-23T02:00:45.5538390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52d77qpx/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5538930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5539434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.5539777Z ok (6.559s) 2022-11-23T02:00:45.5539921Z 2022-11-23T02:00:45.5540269Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5540612Z Ran 1 test in 6.560s 2022-11-23T02:00:45.5540768Z 2022-11-23T02:00:45.5540853Z OK 2022-11-23T02:00:45.5540981Z 2022-11-23T02:00:45.5541098Z Generating XML reports... 2022-11-23T02:00:45.5541757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014040.xml 2022-11-23T02:00:45.5542463Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5543118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5543592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5544207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5544692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5544923Z 2022-11-23T02:00:45.5545024Z Running tests... 2022-11-23T02:00:45.5545453Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5545912Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:00:45.5546479Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6041 2022-11-23T02:00:45.5547027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6042 2022-11-23T02:00:45.5547554Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5548261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5548730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5549351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5549831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5550301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5550965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5551424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5552041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5552573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5553034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5553755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5554497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5555039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5555513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5556025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo602liqr 2022-11-23T02:00:45.5556591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo602liqr/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5557146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8f8x9b9 2022-11-23T02:00:45.5557702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8f8x9b9/_remote_module_non_scriptable.py 2022-11-23T02:00:45.5558093Z ok (6.758s) 2022-11-23T02:00:45.5558280Z 2022-11-23T02:00:45.5558574Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5558899Z Ran 1 test in 6.758s 2022-11-23T02:00:45.5559057Z 2022-11-23T02:00:45.5559138Z OK 2022-11-23T02:00:45.5559265Z 2022-11-23T02:00:45.5559381Z Generating XML reports... 2022-11-23T02:00:45.5560038Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014049.xml 2022-11-23T02:00:45.5560810Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5561488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5561958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5562565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5563056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5563289Z 2022-11-23T02:00:45.5563387Z Running tests... 2022-11-23T02:00:45.5563824Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5564321Z test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.5564613Z 2022-11-23T02:00:45.5564897Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5565227Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5565372Z 2022-11-23T02:00:45.5565466Z OK (skipped=1) 2022-11-23T02:00:45.5565611Z 2022-11-23T02:00:45.5565733Z Generating XML reports... 2022-11-23T02:00:45.5566389Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014059.xml 2022-11-23T02:00:45.5567093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5567758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5568219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5568839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5569315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5569545Z 2022-11-23T02:00:45.5569643Z Running tests... 2022-11-23T02:00:45.5570073Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5570613Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:00:45.5570999Z 2022-11-23T02:00:45.5571283Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5571611Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5571768Z 2022-11-23T02:00:45.5571863Z OK (skipped=1) 2022-11-23T02:00:45.5572019Z 2022-11-23T02:00:45.5572122Z Generating XML reports... 2022-11-23T02:00:45.5572784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014102.xml 2022-11-23T02:00:45.5573497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5574167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5574641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5575267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5575766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5575998Z 2022-11-23T02:00:45.5576100Z Running tests... 2022-11-23T02:00:45.5576524Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5577124Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:00:45.5577460Z 2022-11-23T02:00:45.5577748Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5578082Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5578240Z 2022-11-23T02:00:45.5578341Z OK (skipped=1) 2022-11-23T02:00:45.5578491Z 2022-11-23T02:00:45.5578607Z Generating XML reports... 2022-11-23T02:00:45.5579251Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014105.xml 2022-11-23T02:00:45.5579961Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5580627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5581095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5581718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5582212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5582444Z 2022-11-23T02:00:45.5582546Z Running tests... 2022-11-23T02:00:45.5582985Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5583506Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:00:45.5583832Z 2022-11-23T02:00:45.5584115Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5584455Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5584615Z 2022-11-23T02:00:45.5584714Z OK (skipped=1) 2022-11-23T02:00:45.5584863Z 2022-11-23T02:00:45.5584980Z Generating XML reports... 2022-11-23T02:00:45.5585640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014108.xml 2022-11-23T02:00:45.5586349Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5587001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5587471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5588096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5588590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5588873Z 2022-11-23T02:00:45.5588974Z Running tests... 2022-11-23T02:00:45.5589408Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5589948Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:00:45.5590280Z 2022-11-23T02:00:45.5590565Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5590888Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5590895Z 2022-11-23T02:00:45.5590992Z OK (skipped=1) 2022-11-23T02:00:45.5590999Z 2022-11-23T02:00:45.5591116Z Generating XML reports... 2022-11-23T02:00:45.5591606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014111.xml 2022-11-23T02:00:45.5591945Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5592345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5592521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5592933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5593171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5593178Z 2022-11-23T02:00:45.5593280Z Running tests... 2022-11-23T02:00:45.5593566Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5593904Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s) 2022-11-23T02:00:45.5593912Z 2022-11-23T02:00:45.5594209Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5594316Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5594322Z 2022-11-23T02:00:45.5594422Z OK (skipped=1) 2022-11-23T02:00:45.5594431Z 2022-11-23T02:00:45.5594551Z Generating XML reports... 2022-11-23T02:00:45.5595040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014114.xml 2022-11-23T02:00:45.5595381Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5595782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5595958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5596351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5596542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5596550Z 2022-11-23T02:00:45.5596652Z Running tests... 2022-11-23T02:00:45.5596944Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5597223Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.5597229Z 2022-11-23T02:00:45.5597510Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5597616Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5597622Z 2022-11-23T02:00:45.5597727Z OK (skipped=1) 2022-11-23T02:00:45.5597733Z 2022-11-23T02:00:45.5597853Z Generating XML reports... 2022-11-23T02:00:45.5598340Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014117.xml 2022-11-23T02:00:45.5598679Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5599080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5599412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5599879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5600071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5600078Z 2022-11-23T02:00:45.5600245Z Running tests... 2022-11-23T02:00:45.5600534Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5600869Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6643 2022-11-23T02:00:45.5601096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6644 2022-11-23T02:00:45.5601376Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5601776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5601953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5602366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5602545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5602856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5603259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5603437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5603847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5604040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5604296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5604733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5605164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5605404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5605630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5605999Z STAGE:2022-11-23 01:41:23 6644:6644 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5606359Z STAGE:2022-11-23 01:41:23 6643:6643 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5606730Z STAGE:2022-11-23 01:41:23 6643:6643 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5607087Z STAGE:2022-11-23 01:41:23 6644:6644 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5607472Z STAGE:2022-11-23 01:41:23 6643:6643 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5607849Z STAGE:2022-11-23 01:41:23 6644:6644 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5608098Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5608341Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5608695Z STAGE:2022-11-23 01:41:23 6643:6643 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5609055Z STAGE:2022-11-23 01:41:23 6643:6643 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5609432Z STAGE:2022-11-23 01:41:23 6643:6643 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5609786Z STAGE:2022-11-23 01:41:23 6644:6644 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5610216Z STAGE:2022-11-23 01:41:23 6644:6644 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5610584Z STAGE:2022-11-23 01:41:23 6644:6644 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5610682Z ok (4.339s) 2022-11-23T02:00:45.5610693Z 2022-11-23T02:00:45.5610980Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5611086Z Ran 1 test in 4.339s 2022-11-23T02:00:45.5611093Z 2022-11-23T02:00:45.5611225Z OK 2022-11-23T02:00:45.5611231Z 2022-11-23T02:00:45.5611350Z Generating XML reports... 2022-11-23T02:00:45.5611847Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014120.xml 2022-11-23T02:00:45.5612188Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5612589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5612770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5613184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5613430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5613437Z 2022-11-23T02:00:45.5613541Z Running tests... 2022-11-23T02:00:45.5613831Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5614179Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6835 2022-11-23T02:00:45.5614403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6836 2022-11-23T02:00:45.5614687Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5615088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5615270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5615683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5615877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5616116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5616516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5616694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5617102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5617300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5617554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5617990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5618421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5618658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5618885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5619249Z STAGE:2022-11-23 01:41:30 6836:6836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5619607Z STAGE:2022-11-23 01:41:30 6835:6835 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5620049Z STAGE:2022-11-23 01:41:31 6836:6836 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5620424Z STAGE:2022-11-23 01:41:31 6836:6836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5620671Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5621034Z STAGE:2022-11-23 01:41:31 6835:6835 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5621407Z STAGE:2022-11-23 01:41:31 6835:6835 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5621643Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5621996Z STAGE:2022-11-23 01:41:31 6836:6836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5622357Z STAGE:2022-11-23 01:41:31 6836:6836 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5622733Z STAGE:2022-11-23 01:41:31 6836:6836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5623087Z STAGE:2022-11-23 01:41:31 6835:6835 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5623490Z STAGE:2022-11-23 01:41:31 6835:6835 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5623849Z STAGE:2022-11-23 01:41:31 6835:6835 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5623944Z ok (4.650s) 2022-11-23T02:00:45.5623952Z 2022-11-23T02:00:45.5624240Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5624346Z Ran 1 test in 4.651s 2022-11-23T02:00:45.5624353Z 2022-11-23T02:00:45.5637713Z OK 2022-11-23T02:00:45.5637729Z 2022-11-23T02:00:45.5637902Z Generating XML reports... 2022-11-23T02:00:45.5638480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014128.xml 2022-11-23T02:00:45.5638830Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5639234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5639411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5639822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5640018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5640026Z 2022-11-23T02:00:45.5640126Z Running tests... 2022-11-23T02:00:45.5640493Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5640777Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.5640784Z 2022-11-23T02:00:45.5641063Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5641171Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5641178Z 2022-11-23T02:00:45.5641273Z OK (skipped=1) 2022-11-23T02:00:45.5641280Z 2022-11-23T02:00:45.5641395Z Generating XML reports... 2022-11-23T02:00:45.5641895Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014135.xml 2022-11-23T02:00:45.5642221Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5642621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5642793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5643202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5643388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5643564Z 2022-11-23T02:00:45.5643660Z Running tests... 2022-11-23T02:00:45.5643946Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5644213Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.5644220Z 2022-11-23T02:00:45.5644505Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5644607Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5644614Z 2022-11-23T02:00:45.5644711Z OK (skipped=1) 2022-11-23T02:00:45.5644718Z 2022-11-23T02:00:45.5644834Z Generating XML reports... 2022-11-23T02:00:45.5645326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014138.xml 2022-11-23T02:00:45.5645663Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5646066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5646247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5646651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5646913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5646921Z 2022-11-23T02:00:45.5647019Z Running tests... 2022-11-23T02:00:45.5647303Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5647648Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7143 2022-11-23T02:00:45.5647864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7144 2022-11-23T02:00:45.5648141Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5648526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5648698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5649105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5649298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5649546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5649937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5650110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5650514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5650704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5650950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5651382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5651815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5652049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5652271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5652632Z STAGE:2022-11-23 01:41:44 7144:7144 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5652983Z STAGE:2022-11-23 01:41:44 7143:7143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5653395Z STAGE:2022-11-23 01:41:44 7144:7144 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5653773Z STAGE:2022-11-23 01:41:44 7144:7144 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5654017Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5654376Z STAGE:2022-11-23 01:41:44 7143:7143 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5654747Z STAGE:2022-11-23 01:41:44 7143:7143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5654979Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5655067Z ok (4.549s) 2022-11-23T02:00:45.5655074Z 2022-11-23T02:00:45.5655350Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5655447Z Ran 1 test in 4.549s 2022-11-23T02:00:45.5655454Z 2022-11-23T02:00:45.5655538Z OK 2022-11-23T02:00:45.5655545Z 2022-11-23T02:00:45.5655657Z Generating XML reports... 2022-11-23T02:00:45.5656149Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014142.xml 2022-11-23T02:00:45.5656528Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5656931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5657102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5657512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5657702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5657710Z 2022-11-23T02:00:45.5657805Z Running tests... 2022-11-23T02:00:45.5658087Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5658436Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7335 2022-11-23T02:00:45.5658655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7336 2022-11-23T02:00:45.5658932Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5659327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5659497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5659902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5660094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5660345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5660736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5660907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5661304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5661490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5661738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5662170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5662599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5662883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5663101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5663456Z STAGE:2022-11-23 01:41:52 7336:7336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5663810Z STAGE:2022-11-23 01:41:52 7335:7335 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5664169Z STAGE:2022-11-23 01:41:52 7336:7336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5664544Z STAGE:2022-11-23 01:41:52 7336:7336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5664789Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5665145Z STAGE:2022-11-23 01:41:52 7335:7335 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5665519Z STAGE:2022-11-23 01:41:52 7335:7335 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5665759Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5665849Z ok (4.549s) 2022-11-23T02:00:45.5665856Z 2022-11-23T02:00:45.5666187Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5666289Z Ran 1 test in 4.550s 2022-11-23T02:00:45.5666296Z 2022-11-23T02:00:45.5666378Z OK 2022-11-23T02:00:45.5666385Z 2022-11-23T02:00:45.5666499Z Generating XML reports... 2022-11-23T02:00:45.5666994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014149.xml 2022-11-23T02:00:45.5667328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5667725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5667891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5668298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5668487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5668494Z 2022-11-23T02:00:45.5668594Z Running tests... 2022-11-23T02:00:45.5668880Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5669879Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.515s) 2022-11-23T02:00:45.5669889Z 2022-11-23T02:00:45.5670167Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5670271Z Ran 1 test in 0.515s 2022-11-23T02:00:45.5670277Z 2022-11-23T02:00:45.5670373Z OK (skipped=1) 2022-11-23T02:00:45.5670380Z 2022-11-23T02:00:45.5670496Z Generating XML reports... 2022-11-23T02:00:45.5670986Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014157.xml 2022-11-23T02:00:45.5671323Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5671719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5671895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5672298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5672484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5672491Z 2022-11-23T02:00:45.5672633Z Running tests... 2022-11-23T02:00:45.5672918Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5673920Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.515s) 2022-11-23T02:00:45.5673929Z 2022-11-23T02:00:45.5674210Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5674313Z Ran 1 test in 0.516s 2022-11-23T02:00:45.5674320Z 2022-11-23T02:00:45.5674417Z OK (skipped=1) 2022-11-23T02:00:45.5674423Z 2022-11-23T02:00:45.5674539Z Generating XML reports... 2022-11-23T02:00:45.5675028Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014200.xml 2022-11-23T02:00:45.5675364Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5675758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5675928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5676377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5676556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5676572Z 2022-11-23T02:00:45.5676659Z Running tests... 2022-11-23T02:00:45.5676943Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5677284Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7643 2022-11-23T02:00:45.5677501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7644 2022-11-23T02:00:45.5677778Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5678176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5678351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5678759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5678952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5679205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5679596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5679763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5680276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5680465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5680718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5681147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5681582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5681814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5682045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5682138Z ok (4.454s) 2022-11-23T02:00:45.5682200Z 2022-11-23T02:00:45.5682494Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5682583Z Ran 1 test in 4.454s 2022-11-23T02:00:45.5682599Z 2022-11-23T02:00:45.5682668Z OK 2022-11-23T02:00:45.5682674Z 2022-11-23T02:00:45.5682789Z Generating XML reports... 2022-11-23T02:00:45.5683285Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014204.xml 2022-11-23T02:00:45.5683618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5684015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5684185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5684589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5684775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5684785Z 2022-11-23T02:00:45.5684881Z Running tests... 2022-11-23T02:00:45.5685163Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5685553Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7829 2022-11-23T02:00:45.5685772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7830 2022-11-23T02:00:45.5686051Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5686452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5686623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5687029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5687219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5687465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5687866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5688037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5688446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5688629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5688865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5689296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5689728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5689959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5690188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5690428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.5690669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.5691098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.5691524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.5691767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:00:45.5692054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:00:45.5692485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:00:45.5692909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:00:45.5693150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T02:00:45.5693385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T02:00:45.5693803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:00:45.5694221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:00:45.5694312Z ok (5.154s) 2022-11-23T02:00:45.5694320Z 2022-11-23T02:00:45.5694605Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5694705Z Ran 1 test in 5.154s 2022-11-23T02:00:45.5694712Z 2022-11-23T02:00:45.5694791Z OK 2022-11-23T02:00:45.5694797Z 2022-11-23T02:00:45.5694956Z Generating XML reports... 2022-11-23T02:00:45.5695458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014212.xml 2022-11-23T02:00:45.5695794Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5696179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5696356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5696766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5696964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5696971Z 2022-11-23T02:00:45.5697070Z Running tests... 2022-11-23T02:00:45.5697356Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5697686Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8050 2022-11-23T02:00:45.5697905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8051 2022-11-23T02:00:45.5698182Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5698575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5698742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5699151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5699338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5699585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5699977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5700147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5700552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5700739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5700986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5701416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5701890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5702117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5702349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5702429Z ok (4.349s) 2022-11-23T02:00:45.5702445Z 2022-11-23T02:00:45.5702714Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5702817Z Ran 1 test in 4.349s 2022-11-23T02:00:45.5702824Z 2022-11-23T02:00:45.5702904Z OK 2022-11-23T02:00:45.5702910Z 2022-11-23T02:00:45.5703024Z Generating XML reports... 2022-11-23T02:00:45.5703518Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014220.xml 2022-11-23T02:00:45.5703855Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5704256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5704428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5704902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5705099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5705106Z 2022-11-23T02:00:45.5705203Z Running tests... 2022-11-23T02:00:45.5705486Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5705954Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:00:45.5705962Z 2022-11-23T02:00:45.5706241Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5706346Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5706353Z 2022-11-23T02:00:45.5706450Z OK (skipped=1) 2022-11-23T02:00:45.5706456Z 2022-11-23T02:00:45.5706569Z Generating XML reports... 2022-11-23T02:00:45.5707057Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014227.xml 2022-11-23T02:00:45.5707389Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5707795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5707969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5708364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5708549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5708571Z 2022-11-23T02:00:45.5708657Z Running tests... 2022-11-23T02:00:45.5708935Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5709394Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.5709404Z 2022-11-23T02:00:45.5709732Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5709831Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5709837Z 2022-11-23T02:00:45.5709935Z OK (skipped=1) 2022-11-23T02:00:45.5709941Z 2022-11-23T02:00:45.5710054Z Generating XML reports... 2022-11-23T02:00:45.5710545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014230.xml 2022-11-23T02:00:45.5710882Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5711276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5711499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5711910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5712104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5712111Z 2022-11-23T02:00:45.5712209Z Running tests... 2022-11-23T02:00:45.5712490Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5712963Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.5712970Z 2022-11-23T02:00:45.5713251Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5713355Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5713362Z 2022-11-23T02:00:45.5713457Z OK (skipped=1) 2022-11-23T02:00:45.5713467Z 2022-11-23T02:00:45.5713581Z Generating XML reports... 2022-11-23T02:00:45.5714070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014233.xml 2022-11-23T02:00:45.5714403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5714837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5715013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5715424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5715613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5715620Z 2022-11-23T02:00:45.5715718Z Running tests... 2022-11-23T02:00:45.5715996Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5716457Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:00:45.5716464Z 2022-11-23T02:00:45.5716741Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5716843Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5716852Z 2022-11-23T02:00:45.5716950Z OK (skipped=1) 2022-11-23T02:00:45.5716956Z 2022-11-23T02:00:45.5717070Z Generating XML reports... 2022-11-23T02:00:45.5717551Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014236.xml 2022-11-23T02:00:45.5717886Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5718280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5718453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5718858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5719047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5719054Z 2022-11-23T02:00:45.5719153Z Running tests... 2022-11-23T02:00:45.5719438Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5719890Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:00:45.5719897Z 2022-11-23T02:00:45.5720237Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5720338Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5720346Z 2022-11-23T02:00:45.5720444Z OK (skipped=1) 2022-11-23T02:00:45.5720450Z 2022-11-23T02:00:45.5720553Z Generating XML reports... 2022-11-23T02:00:45.5721036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014240.xml 2022-11-23T02:00:45.5721422Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5721815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5721990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5722396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5722581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5722587Z 2022-11-23T02:00:45.5722690Z Running tests... 2022-11-23T02:00:45.5722972Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5723419Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:00:45.5723429Z 2022-11-23T02:00:45.5723707Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5723810Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5723817Z 2022-11-23T02:00:45.5723912Z OK (skipped=1) 2022-11-23T02:00:45.5723919Z 2022-11-23T02:00:45.5724035Z Generating XML reports... 2022-11-23T02:00:45.5724566Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014243.xml 2022-11-23T02:00:45.5724899Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5725292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5725463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5725862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5726048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5726055Z 2022-11-23T02:00:45.5726150Z Running tests... 2022-11-23T02:00:45.5726435Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5726883Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:00:45.5726905Z 2022-11-23T02:00:45.5727175Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5727278Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5727284Z 2022-11-23T02:00:45.5727379Z OK (skipped=1) 2022-11-23T02:00:45.5727385Z 2022-11-23T02:00:45.5727498Z Generating XML reports... 2022-11-23T02:00:45.5727983Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014246.xml 2022-11-23T02:00:45.5728315Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5728707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5728875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5729288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5729473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5729480Z 2022-11-23T02:00:45.5729580Z Running tests... 2022-11-23T02:00:45.5729860Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5730310Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.5730317Z 2022-11-23T02:00:45.5730596Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5730696Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5730753Z 2022-11-23T02:00:45.5730851Z OK (skipped=1) 2022-11-23T02:00:45.5730857Z 2022-11-23T02:00:45.5730975Z Generating XML reports... 2022-11-23T02:00:45.5731458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014249.xml 2022-11-23T02:00:45.5731793Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5732190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5732363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5732767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5732956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5732963Z 2022-11-23T02:00:45.5733048Z Running tests... 2022-11-23T02:00:45.5733326Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5733756Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.5733763Z 2022-11-23T02:00:45.5734036Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5734183Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5734191Z 2022-11-23T02:00:45.5734291Z OK (skipped=1) 2022-11-23T02:00:45.5734297Z 2022-11-23T02:00:45.5734417Z Generating XML reports... 2022-11-23T02:00:45.5734905Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014252.xml 2022-11-23T02:00:45.5735240Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5735636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5735810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5736214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5736401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5736408Z 2022-11-23T02:00:45.5736510Z Running tests... 2022-11-23T02:00:45.5736794Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5737110Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5737118Z 2022-11-23T02:00:45.5737395Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5737498Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5737505Z 2022-11-23T02:00:45.5737599Z OK (skipped=1) 2022-11-23T02:00:45.5737606Z 2022-11-23T02:00:45.5737719Z Generating XML reports... 2022-11-23T02:00:45.5738208Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014255.xml 2022-11-23T02:00:45.5738541Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5738937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5739097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5739507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5739696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5739703Z 2022-11-23T02:00:45.5739798Z Running tests... 2022-11-23T02:00:45.5740078Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5740510Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:00:45.5740563Z 2022-11-23T02:00:45.5740839Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5740942Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5740949Z 2022-11-23T02:00:45.5741043Z OK (skipped=1) 2022-11-23T02:00:45.5741049Z 2022-11-23T02:00:45.5741166Z Generating XML reports... 2022-11-23T02:00:45.5741654Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014258.xml 2022-11-23T02:00:45.5741988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5742384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5742556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5742961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5743150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5743157Z 2022-11-23T02:00:45.5743255Z Running tests... 2022-11-23T02:00:45.5743536Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5744026Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:00:45.5744034Z 2022-11-23T02:00:45.5744312Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5744415Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5744422Z 2022-11-23T02:00:45.5744514Z OK (skipped=1) 2022-11-23T02:00:45.5744519Z 2022-11-23T02:00:45.5744632Z Generating XML reports... 2022-11-23T02:00:45.5745106Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014301.xml 2022-11-23T02:00:45.5745437Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5745833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5746004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5746415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5746603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5746610Z 2022-11-23T02:00:45.5746704Z Running tests... 2022-11-23T02:00:45.5746987Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5747417Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:00:45.5747424Z 2022-11-23T02:00:45.5747701Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5747807Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5747814Z 2022-11-23T02:00:45.5747909Z OK (skipped=1) 2022-11-23T02:00:45.5747919Z 2022-11-23T02:00:45.5748034Z Generating XML reports... 2022-11-23T02:00:45.5748514Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014304.xml 2022-11-23T02:00:45.5748854Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5749254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5749426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5749836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5750024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5750031Z 2022-11-23T02:00:45.5750129Z Running tests... 2022-11-23T02:00:45.5750461Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5750762Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5750769Z 2022-11-23T02:00:45.5751049Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5751140Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5751156Z 2022-11-23T02:00:45.5751242Z OK (skipped=1) 2022-11-23T02:00:45.5751247Z 2022-11-23T02:00:45.5751363Z Generating XML reports... 2022-11-23T02:00:45.5751846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014307.xml 2022-11-23T02:00:45.5752182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5752580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5752755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5753160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5753349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5753356Z 2022-11-23T02:00:45.5753497Z Running tests... 2022-11-23T02:00:45.5753784Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5754060Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5754067Z 2022-11-23T02:00:45.5754345Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5754445Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5754451Z 2022-11-23T02:00:45.5754544Z OK (skipped=1) 2022-11-23T02:00:45.5754551Z 2022-11-23T02:00:45.5754665Z Generating XML reports... 2022-11-23T02:00:45.5755152Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014310.xml 2022-11-23T02:00:45.5755484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5755885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5756054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5756458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5756643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5756650Z 2022-11-23T02:00:45.5756746Z Running tests... 2022-11-23T02:00:45.5757015Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5757295Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.5757305Z 2022-11-23T02:00:45.5757582Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5757684Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5757691Z 2022-11-23T02:00:45.5757787Z OK (skipped=1) 2022-11-23T02:00:45.5757793Z 2022-11-23T02:00:45.5757909Z Generating XML reports... 2022-11-23T02:00:45.5758392Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014313.xml 2022-11-23T02:00:45.5758725Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5759120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5759290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5759689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5759925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5759931Z 2022-11-23T02:00:45.5760028Z Running tests... 2022-11-23T02:00:45.5760381Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5760676Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5760683Z 2022-11-23T02:00:45.5760962Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5761063Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5761069Z 2022-11-23T02:00:45.5761165Z OK (skipped=1) 2022-11-23T02:00:45.5761171Z 2022-11-23T02:00:45.5761286Z Generating XML reports... 2022-11-23T02:00:45.5761780Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014316.xml 2022-11-23T02:00:45.5762115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5762512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5762685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5763131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5763317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5763323Z 2022-11-23T02:00:45.5763422Z Running tests... 2022-11-23T02:00:45.5763700Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5763980Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5763987Z 2022-11-23T02:00:45.5764261Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5764365Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5764372Z 2022-11-23T02:00:45.5764472Z OK (skipped=1) 2022-11-23T02:00:45.5764478Z 2022-11-23T02:00:45.5764594Z Generating XML reports... 2022-11-23T02:00:45.5765080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014319.xml 2022-11-23T02:00:45.5765414Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5765808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5765980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5766389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5766577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5766584Z 2022-11-23T02:00:45.5766683Z Running tests... 2022-11-23T02:00:45.5766968Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5767241Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5767248Z 2022-11-23T02:00:45.5767523Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5767627Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5767634Z 2022-11-23T02:00:45.5767732Z OK (skipped=1) 2022-11-23T02:00:45.5767738Z 2022-11-23T02:00:45.5767853Z Generating XML reports... 2022-11-23T02:00:45.5768323Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014322.xml 2022-11-23T02:00:45.5768656Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5769052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5769280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5769690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5769880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5769887Z 2022-11-23T02:00:45.5769988Z Running tests... 2022-11-23T02:00:45.5770267Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5770541Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.5770548Z 2022-11-23T02:00:45.5770823Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5770922Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5770929Z 2022-11-23T02:00:45.5771027Z OK (skipped=1) 2022-11-23T02:00:45.5771034Z 2022-11-23T02:00:45.5771149Z Generating XML reports... 2022-11-23T02:00:45.5771633Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014325.xml 2022-11-23T02:00:45.5771971Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5772409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5772581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5772990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5773181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5773188Z 2022-11-23T02:00:45.5773287Z Running tests... 2022-11-23T02:00:45.5773562Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5773843Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5773853Z 2022-11-23T02:00:45.5774131Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5774221Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5774238Z 2022-11-23T02:00:45.5774322Z OK (skipped=1) 2022-11-23T02:00:45.5774329Z 2022-11-23T02:00:45.5774441Z Generating XML reports... 2022-11-23T02:00:45.5774921Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014328.xml 2022-11-23T02:00:45.5775251Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5775646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5775818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5776221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5776409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5776416Z 2022-11-23T02:00:45.5776513Z Running tests... 2022-11-23T02:00:45.5776792Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5777069Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5777075Z 2022-11-23T02:00:45.5777355Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5777456Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5777462Z 2022-11-23T02:00:45.5777561Z OK (skipped=1) 2022-11-23T02:00:45.5777567Z 2022-11-23T02:00:45.5777681Z Generating XML reports... 2022-11-23T02:00:45.5778166Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014332.xml 2022-11-23T02:00:45.5778499Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5778967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5779138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5779551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5779739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5779745Z 2022-11-23T02:00:45.5779841Z Running tests... 2022-11-23T02:00:45.5780110Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5780378Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.5780385Z 2022-11-23T02:00:45.5780657Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5780757Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5780767Z 2022-11-23T02:00:45.5780863Z OK (skipped=1) 2022-11-23T02:00:45.5780869Z 2022-11-23T02:00:45.5780983Z Generating XML reports... 2022-11-23T02:00:45.5781464Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014335.xml 2022-11-23T02:00:45.5781837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5782236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5782407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5782817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5783007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5783013Z 2022-11-23T02:00:45.5783113Z Running tests... 2022-11-23T02:00:45.5783395Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5783663Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.5783669Z 2022-11-23T02:00:45.5783948Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5784051Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5784061Z 2022-11-23T02:00:45.5784153Z OK (skipped=1) 2022-11-23T02:00:45.5784159Z 2022-11-23T02:00:45.5784273Z Generating XML reports... 2022-11-23T02:00:45.5784761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014338.xml 2022-11-23T02:00:45.5785093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5785482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5785641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5786053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5786241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5786247Z 2022-11-23T02:00:45.5786345Z Running tests... 2022-11-23T02:00:45.5786629Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5786916Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s) 2022-11-23T02:00:45.5786923Z 2022-11-23T02:00:45.5787199Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5787303Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5787309Z 2022-11-23T02:00:45.5787406Z OK (skipped=1) 2022-11-23T02:00:45.5787413Z 2022-11-23T02:00:45.5787527Z Generating XML reports... 2022-11-23T02:00:45.5788020Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014341.xml 2022-11-23T02:00:45.5788401Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5788795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5788967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5789372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5789560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5789567Z 2022-11-23T02:00:45.5789662Z Running tests... 2022-11-23T02:00:45.5789942Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5790243Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s) 2022-11-23T02:00:45.5790254Z 2022-11-23T02:00:45.5790526Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5790627Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5790634Z 2022-11-23T02:00:45.5790729Z OK (skipped=1) 2022-11-23T02:00:45.5790735Z 2022-11-23T02:00:45.5790852Z Generating XML reports... 2022-11-23T02:00:45.5791366Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014344.xml 2022-11-23T02:00:45.5791707Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5792101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5792269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5792677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5792863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5792874Z 2022-11-23T02:00:45.5792976Z Running tests... 2022-11-23T02:00:45.5793259Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5793534Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5793541Z 2022-11-23T02:00:45.5793814Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5793916Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5793922Z 2022-11-23T02:00:45.5794017Z OK (skipped=1) 2022-11-23T02:00:45.5794023Z 2022-11-23T02:00:45.5794137Z Generating XML reports... 2022-11-23T02:00:45.5794621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014347.xml 2022-11-23T02:00:45.5794955Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5795356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5795527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5795936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5796126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5796132Z 2022-11-23T02:00:45.5796229Z Running tests... 2022-11-23T02:00:45.5796513Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5796847Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9806 2022-11-23T02:00:45.5797065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9807 2022-11-23T02:00:45.5797332Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5797773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5797946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5798361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5798552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5798806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5799203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5799375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5799782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5799973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5800280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5800767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5801203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5801437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5801667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5801763Z ok (4.457s) 2022-11-23T02:00:45.5801770Z 2022-11-23T02:00:45.5802055Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5802161Z Ran 1 test in 4.457s 2022-11-23T02:00:45.5802171Z 2022-11-23T02:00:45.5802256Z OK 2022-11-23T02:00:45.5802262Z 2022-11-23T02:00:45.5802375Z Generating XML reports... 2022-11-23T02:00:45.5802867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014350.xml 2022-11-23T02:00:45.5803207Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5803782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5803959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5804366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5804556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5804563Z 2022-11-23T02:00:45.5804662Z Running tests... 2022-11-23T02:00:45.5804945Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5805220Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5805226Z 2022-11-23T02:00:45.5805505Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5805610Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5805617Z 2022-11-23T02:00:45.5805714Z OK (skipped=1) 2022-11-23T02:00:45.5805720Z 2022-11-23T02:00:45.5805836Z Generating XML reports... 2022-11-23T02:00:45.5806326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014357.xml 2022-11-23T02:00:45.5806658Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5807054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5807224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5807686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5807877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5807884Z 2022-11-23T02:00:45.5807979Z Running tests... 2022-11-23T02:00:45.5808257Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5808531Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5808539Z 2022-11-23T02:00:45.5808817Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5808917Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5808924Z 2022-11-23T02:00:45.5809016Z OK (skipped=1) 2022-11-23T02:00:45.5809023Z 2022-11-23T02:00:45.5809126Z Generating XML reports... 2022-11-23T02:00:45.5809609Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014400.xml 2022-11-23T02:00:45.5809946Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5810346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5810593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5811006Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5811196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5811203Z 2022-11-23T02:00:45.5811303Z Running tests... 2022-11-23T02:00:45.5811586Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5811861Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.5811871Z 2022-11-23T02:00:45.5812146Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5812246Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5812253Z 2022-11-23T02:00:45.5812348Z OK (skipped=1) 2022-11-23T02:00:45.5812355Z 2022-11-23T02:00:45.5812471Z Generating XML reports... 2022-11-23T02:00:45.5812962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014404.xml 2022-11-23T02:00:45.5813298Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5813698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5813874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5814278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5814469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5814479Z 2022-11-23T02:00:45.5814579Z Running tests... 2022-11-23T02:00:45.5814856Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5815180Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10166 2022-11-23T02:00:45.5815404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10167 2022-11-23T02:00:45.5815684Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5816080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5816249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5816655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5816889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5817139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5817537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5817711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5818118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5818300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5818549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5818983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5819414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5819646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5819916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5820289Z STAGE:2022-11-23 01:44:09 10167:10167 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5820654Z STAGE:2022-11-23 01:44:09 10166:10166 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5821018Z STAGE:2022-11-23 01:44:10 10166:10166 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5821372Z STAGE:2022-11-23 01:44:10 10167:10167 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5821754Z STAGE:2022-11-23 01:44:10 10166:10166 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5822135Z STAGE:2022-11-23 01:44:10 10167:10167 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5822381Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5822613Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5822969Z STAGE:2022-11-23 01:44:10 10167:10167 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5823316Z STAGE:2022-11-23 01:44:10 10166:10166 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5823680Z STAGE:2022-11-23 01:44:10 10167:10167 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5824058Z STAGE:2022-11-23 01:44:10 10167:10167 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5824420Z STAGE:2022-11-23 01:44:10 10166:10166 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5824801Z STAGE:2022-11-23 01:44:10 10166:10166 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5825152Z STAGE:2022-11-23 01:44:10 10167:10167 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5825517Z STAGE:2022-11-23 01:44:10 10167:10167 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5825893Z STAGE:2022-11-23 01:44:10 10167:10167 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5826241Z STAGE:2022-11-23 01:44:10 10166:10166 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5826602Z STAGE:2022-11-23 01:44:10 10166:10166 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5826978Z STAGE:2022-11-23 01:44:10 10166:10166 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5827073Z ok (4.658s) 2022-11-23T02:00:45.5827126Z 2022-11-23T02:00:45.5827415Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5827520Z Ran 1 test in 4.658s 2022-11-23T02:00:45.5827526Z 2022-11-23T02:00:45.5827609Z OK 2022-11-23T02:00:45.5827615Z 2022-11-23T02:00:45.5827726Z Generating XML reports... 2022-11-23T02:00:45.5828223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014407.xml 2022-11-23T02:00:45.5828563Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5828964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5829137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5829535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5829721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5829743Z 2022-11-23T02:00:45.5829831Z Running tests... 2022-11-23T02:00:45.5830114Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5830498Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10358 2022-11-23T02:00:45.5830724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10359 2022-11-23T02:00:45.5830999Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5831401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5831579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5831984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5832174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5832426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5832827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5833002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5833409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5833595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5833841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5834277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5834710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5834942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5835171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5835536Z STAGE:2022-11-23 01:44:17 10359:10359 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5835891Z STAGE:2022-11-23 01:44:17 10358:10358 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5836241Z STAGE:2022-11-23 01:44:17 10358:10358 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5836617Z STAGE:2022-11-23 01:44:17 10358:10358 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5836982Z STAGE:2022-11-23 01:44:17 10359:10359 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5837270Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5837646Z STAGE:2022-11-23 01:44:17 10359:10359 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5837891Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5838247Z STAGE:2022-11-23 01:44:17 10358:10358 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5838596Z STAGE:2022-11-23 01:44:17 10359:10359 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5838957Z STAGE:2022-11-23 01:44:17 10358:10358 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5839332Z STAGE:2022-11-23 01:44:17 10358:10358 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5839690Z STAGE:2022-11-23 01:44:17 10359:10359 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5840065Z STAGE:2022-11-23 01:44:17 10359:10359 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5840476Z STAGE:2022-11-23 01:44:17 10358:10358 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5840887Z STAGE:2022-11-23 01:44:17 10359:10359 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5841254Z STAGE:2022-11-23 01:44:17 10358:10358 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5841631Z STAGE:2022-11-23 01:44:17 10358:10358 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5841988Z STAGE:2022-11-23 01:44:17 10359:10359 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5842361Z STAGE:2022-11-23 01:44:17 10359:10359 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5842456Z ok (4.661s) 2022-11-23T02:00:45.5842466Z 2022-11-23T02:00:45.5842751Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5842854Z Ran 1 test in 4.662s 2022-11-23T02:00:45.5842861Z 2022-11-23T02:00:45.5842945Z OK 2022-11-23T02:00:45.5842951Z 2022-11-23T02:00:45.5843063Z Generating XML reports... 2022-11-23T02:00:45.5843548Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014414.xml 2022-11-23T02:00:45.5843882Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5844277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5844450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5844856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5845053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5845060Z 2022-11-23T02:00:45.5845161Z Running tests... 2022-11-23T02:00:45.5845441Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5845797Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10550 2022-11-23T02:00:45.5846022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10551 2022-11-23T02:00:45.5846306Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5846700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5846874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5847279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5847517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5847762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5848163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5848342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5848753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5848944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5849194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5849629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5850058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5850292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5850579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5850949Z STAGE:2022-11-23 01:44:25 10550:10550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5851306Z STAGE:2022-11-23 01:44:25 10551:10551 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5851674Z STAGE:2022-11-23 01:44:25 10550:10550 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5852050Z STAGE:2022-11-23 01:44:25 10550:10550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5852410Z STAGE:2022-11-23 01:44:25 10551:10551 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5852654Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5853027Z STAGE:2022-11-23 01:44:25 10551:10551 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5853264Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5853615Z STAGE:2022-11-23 01:44:25 10550:10550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5853961Z STAGE:2022-11-23 01:44:25 10551:10551 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5854322Z STAGE:2022-11-23 01:44:25 10550:10550 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5854681Z STAGE:2022-11-23 01:44:25 10551:10551 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5855055Z STAGE:2022-11-23 01:44:25 10550:10550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5855434Z STAGE:2022-11-23 01:44:25 10551:10551 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5855790Z STAGE:2022-11-23 01:44:25 10550:10550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5856141Z STAGE:2022-11-23 01:44:25 10551:10551 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5856500Z STAGE:2022-11-23 01:44:25 10551:10551 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5856859Z STAGE:2022-11-23 01:44:25 10550:10550 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5857230Z STAGE:2022-11-23 01:44:25 10551:10551 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5857607Z STAGE:2022-11-23 01:44:25 10550:10550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5857745Z ok (4.279s) 2022-11-23T02:00:45.5857754Z 2022-11-23T02:00:45.5858038Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5858131Z Ran 1 test in 4.280s 2022-11-23T02:00:45.5858149Z 2022-11-23T02:00:45.5858218Z OK 2022-11-23T02:00:45.5858225Z 2022-11-23T02:00:45.5858339Z Generating XML reports... 2022-11-23T02:00:45.5858838Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014422.xml 2022-11-23T02:00:45.5859175Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5859574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5859747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5860154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5860349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5860356Z 2022-11-23T02:00:45.5860459Z Running tests... 2022-11-23T02:00:45.5860743Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5861041Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:00:45.5861049Z 2022-11-23T02:00:45.5861326Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5861426Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5861433Z 2022-11-23T02:00:45.5861531Z OK (skipped=1) 2022-11-23T02:00:45.5861537Z 2022-11-23T02:00:45.5861653Z Generating XML reports... 2022-11-23T02:00:45.5862144Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014430.xml 2022-11-23T02:00:45.5862483Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5862882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5863056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5863462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5863655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5863662Z 2022-11-23T02:00:45.5863748Z Running tests... 2022-11-23T02:00:45.5864033Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5864298Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:00:45.5864305Z 2022-11-23T02:00:45.5864582Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5864684Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5864690Z 2022-11-23T02:00:45.5864789Z OK (skipped=1) 2022-11-23T02:00:45.5864798Z 2022-11-23T02:00:45.5864914Z Generating XML reports... 2022-11-23T02:00:45.5865399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014433.xml 2022-11-23T02:00:45.5865735Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5866135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5866308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5866720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5866904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5866911Z 2022-11-23T02:00:45.5867012Z Running tests... 2022-11-23T02:00:45.5867289Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5867662Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10858 2022-11-23T02:00:45.5867881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10859 2022-11-23T02:00:45.5868159Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5868562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5868736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5869142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5869329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5869578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5869963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5870134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5870585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5870776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5871022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5871457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5871890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5872121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5872356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5872488Z skip: Test skipped for ROCm (3.731s) 2022-11-23T02:00:45.5872495Z 2022-11-23T02:00:45.5872778Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5872885Z Ran 1 test in 3.731s 2022-11-23T02:00:45.5872892Z 2022-11-23T02:00:45.5872987Z OK (skipped=1) 2022-11-23T02:00:45.5872994Z 2022-11-23T02:00:45.5873115Z Generating XML reports... 2022-11-23T02:00:45.5873604Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014436.xml 2022-11-23T02:00:45.5873942Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5874337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5874513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5874921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5875108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5875115Z 2022-11-23T02:00:45.5875220Z Running tests... 2022-11-23T02:00:45.5875500Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5875826Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11037 2022-11-23T02:00:45.5876041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11038 2022-11-23T02:00:45.5876316Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5876709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5876933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5877344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5877534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5877781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5878172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5878344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5878745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5878932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5879179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5879608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5880094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5880383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5880606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5880733Z skip: Test skipped for ROCm (3.969s) 2022-11-23T02:00:45.5880740Z 2022-11-23T02:00:45.5881035Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5881141Z Ran 1 test in 3.969s 2022-11-23T02:00:45.5881147Z 2022-11-23T02:00:45.5881246Z OK (skipped=1) 2022-11-23T02:00:45.5881252Z 2022-11-23T02:00:45.5881374Z Generating XML reports... 2022-11-23T02:00:45.5881870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014443.xml 2022-11-23T02:00:45.5882206Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5882607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5882782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5883195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5883386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5883393Z 2022-11-23T02:00:45.5883491Z Running tests... 2022-11-23T02:00:45.5883764Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5884032Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:00:45.5884055Z 2022-11-23T02:00:45.5884323Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5884425Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5884432Z 2022-11-23T02:00:45.5884529Z OK (skipped=1) 2022-11-23T02:00:45.5884537Z 2022-11-23T02:00:45.5884653Z Generating XML reports... 2022-11-23T02:00:45.5885138Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014450.xml 2022-11-23T02:00:45.5885471Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5885869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5886042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5886451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5886698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5886705Z 2022-11-23T02:00:45.5886801Z Running tests... 2022-11-23T02:00:45.5887083Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5887425Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11274 2022-11-23T02:00:45.5887643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11275 2022-11-23T02:00:45.5887923Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5888323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5888495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5888903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5889095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5889394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5889793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5889955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5890369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5890559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5890810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5891245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5891677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5891911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5892141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5892274Z skip: Test skipped for ROCm (3.773s) 2022-11-23T02:00:45.5892280Z 2022-11-23T02:00:45.5892568Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5892670Z Ran 1 test in 3.773s 2022-11-23T02:00:45.5892676Z 2022-11-23T02:00:45.5892772Z OK (skipped=1) 2022-11-23T02:00:45.5892778Z 2022-11-23T02:00:45.5892893Z Generating XML reports... 2022-11-23T02:00:45.5893383Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014453.xml 2022-11-23T02:00:45.5893718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5894117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5894293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5894704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5894894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5894901Z 2022-11-23T02:00:45.5895001Z Running tests... 2022-11-23T02:00:45.5895285Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5895553Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:00:45.5895561Z 2022-11-23T02:00:45.5895892Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5895984Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5896001Z 2022-11-23T02:00:45.5896085Z OK (skipped=1) 2022-11-23T02:00:45.5896091Z 2022-11-23T02:00:45.5896210Z Generating XML reports... 2022-11-23T02:00:45.5896705Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014500.xml 2022-11-23T02:00:45.5897045Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5897442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5897615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5898027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5898222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5898232Z 2022-11-23T02:00:45.5898330Z Running tests... 2022-11-23T02:00:45.5898612Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5898994Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11511 2022-11-23T02:00:45.5899215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11512 2022-11-23T02:00:45.5899488Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5899889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5900066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5900474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5900667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5900915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5901314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5901492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5901899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5902076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5902325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5902762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5903193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5903430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5903654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5903814Z skip: Skipped due to small world size. (3.956s) 2022-11-23T02:00:45.5903821Z 2022-11-23T02:00:45.5904109Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5904215Z Ran 1 test in 3.956s 2022-11-23T02:00:45.5904222Z 2022-11-23T02:00:45.5904321Z OK (skipped=1) 2022-11-23T02:00:45.5904327Z 2022-11-23T02:00:45.5904446Z Generating XML reports... 2022-11-23T02:00:45.5904942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014503.xml 2022-11-23T02:00:45.5905277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5905738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5905912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5906482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5906670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5906677Z 2022-11-23T02:00:45.5906778Z Running tests... 2022-11-23T02:00:45.5907065Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5907363Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:00:45.5907370Z 2022-11-23T02:00:45.5907651Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5907760Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5907767Z 2022-11-23T02:00:45.5907869Z OK (skipped=1) 2022-11-23T02:00:45.5907875Z 2022-11-23T02:00:45.5907995Z Generating XML reports... 2022-11-23T02:00:45.5908474Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014510.xml 2022-11-23T02:00:45.5908860Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5909262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5909436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5909843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5910033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5910040Z 2022-11-23T02:00:45.5910137Z Running tests... 2022-11-23T02:00:45.5910423Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5910730Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:00:45.5910738Z 2022-11-23T02:00:45.5911023Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5911130Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5911137Z 2022-11-23T02:00:45.5911235Z OK (skipped=1) 2022-11-23T02:00:45.5911241Z 2022-11-23T02:00:45.5911357Z Generating XML reports... 2022-11-23T02:00:45.5911842Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014513.xml 2022-11-23T02:00:45.5912176Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5912576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5912757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5913163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5913354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5913363Z 2022-11-23T02:00:45.5913538Z Running tests... 2022-11-23T02:00:45.5913822Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5914847Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.555s) 2022-11-23T02:00:45.5914856Z 2022-11-23T02:00:45.5915131Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5915277Z Ran 1 test in 0.555s 2022-11-23T02:00:45.5915284Z 2022-11-23T02:00:45.5915377Z OK (skipped=1) 2022-11-23T02:00:45.5915383Z 2022-11-23T02:00:45.5915487Z Generating XML reports... 2022-11-23T02:00:45.5915982Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014516.xml 2022-11-23T02:00:45.5916317Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5916717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5916886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5917294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5917485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5917495Z 2022-11-23T02:00:45.5917591Z Running tests... 2022-11-23T02:00:45.5917871Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5918278Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11864 2022-11-23T02:00:45.5918506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11865 2022-11-23T02:00:45.5918788Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5919188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5919364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5919773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5919962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5920270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5920673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5920848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5921260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5921451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5921701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5922139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5922573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5922795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5923021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5923392Z STAGE:2022-11-23 01:45:22 11865:11865 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5923746Z STAGE:2022-11-23 01:45:22 11864:11864 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.5924111Z STAGE:2022-11-23 01:45:23 11865:11865 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5924493Z STAGE:2022-11-23 01:45:23 11865:11865 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5924861Z STAGE:2022-11-23 01:45:23 11864:11864 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.5925187Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5925563Z STAGE:2022-11-23 01:45:23 11864:11864 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.5925803Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.5925897Z ok (4.868s) 2022-11-23T02:00:45.5925904Z 2022-11-23T02:00:45.5926196Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5926301Z Ran 1 test in 4.868s 2022-11-23T02:00:45.5926308Z 2022-11-23T02:00:45.5926390Z OK 2022-11-23T02:00:45.5926397Z 2022-11-23T02:00:45.5926515Z Generating XML reports... 2022-11-23T02:00:45.5927008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014519.xml 2022-11-23T02:00:45.5927349Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5927755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5927931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5928391Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5928586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5928593Z 2022-11-23T02:00:45.5928693Z Running tests... 2022-11-23T02:00:45.5928968Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5929278Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:00:45.5929285Z 2022-11-23T02:00:45.5929562Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5929669Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5929679Z 2022-11-23T02:00:45.5929778Z OK (skipped=1) 2022-11-23T02:00:45.5929784Z 2022-11-23T02:00:45.5929902Z Generating XML reports... 2022-11-23T02:00:45.5930387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014527.xml 2022-11-23T02:00:45.5930729Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5931126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5931300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5931708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5931900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5931907Z 2022-11-23T02:00:45.5932006Z Running tests... 2022-11-23T02:00:45.5932293Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5933343Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.630s) 2022-11-23T02:00:45.5933353Z 2022-11-23T02:00:45.5933633Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5933734Z Ran 1 test in 0.630s 2022-11-23T02:00:45.5933741Z 2022-11-23T02:00:45.5933838Z OK (skipped=1) 2022-11-23T02:00:45.5933844Z 2022-11-23T02:00:45.5933964Z Generating XML reports... 2022-11-23T02:00:45.5934453Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014530.xml 2022-11-23T02:00:45.5934840Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5935237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5935410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5935820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5936010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5936018Z 2022-11-23T02:00:45.5936122Z Running tests... 2022-11-23T02:00:45.5936392Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5936698Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:00:45.5936706Z 2022-11-23T02:00:45.5936989Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5937095Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5937102Z 2022-11-23T02:00:45.5937204Z OK (skipped=1) 2022-11-23T02:00:45.5937210Z 2022-11-23T02:00:45.5937324Z Generating XML reports... 2022-11-23T02:00:45.5937855Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014534.xml 2022-11-23T02:00:45.5938195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5938594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5938768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5939175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5939363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5939373Z 2022-11-23T02:00:45.5939473Z Running tests... 2022-11-23T02:00:45.5939752Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5940113Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12234 2022-11-23T02:00:45.5940339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12235 2022-11-23T02:00:45.5940618Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5941014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5941187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5941596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5941790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5942040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5942431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5942594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5943001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5943191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5943442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5943879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5944309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5944594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5944824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5944986Z skip: Skipped due to small world size. (3.853s) 2022-11-23T02:00:45.5944994Z 2022-11-23T02:00:45.5945279Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5945385Z Ran 1 test in 3.853s 2022-11-23T02:00:45.5945391Z 2022-11-23T02:00:45.5945487Z OK (skipped=1) 2022-11-23T02:00:45.5945493Z 2022-11-23T02:00:45.5945610Z Generating XML reports... 2022-11-23T02:00:45.5946100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014537.xml 2022-11-23T02:00:45.5946434Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5946836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5947011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5947463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5947656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5947663Z 2022-11-23T02:00:45.5947764Z Running tests... 2022-11-23T02:00:45.5948050Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5948349Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:00:45.5948356Z 2022-11-23T02:00:45.5948634Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5948725Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5948751Z 2022-11-23T02:00:45.5948838Z OK (skipped=1) 2022-11-23T02:00:45.5948844Z 2022-11-23T02:00:45.5948961Z Generating XML reports... 2022-11-23T02:00:45.5949448Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014544.xml 2022-11-23T02:00:45.5949790Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5950186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5950359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5950771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5950961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5950968Z 2022-11-23T02:00:45.5951070Z Running tests... 2022-11-23T02:00:45.5951355Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5951667Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:00:45.5951673Z 2022-11-23T02:00:45.5951950Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5952060Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5952067Z 2022-11-23T02:00:45.5952164Z OK (skipped=1) 2022-11-23T02:00:45.5952170Z 2022-11-23T02:00:45.5952286Z Generating XML reports... 2022-11-23T02:00:45.5952771Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014547.xml 2022-11-23T02:00:45.5953105Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5953502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5953720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5954130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5954320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5954326Z 2022-11-23T02:00:45.5954430Z Running tests... 2022-11-23T02:00:45.5954698Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5955056Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12529 2022-11-23T02:00:45.5955278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12530 2022-11-23T02:00:45.5955555Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5955947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5956126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5956534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5956768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5957019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5957416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5957588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5957996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5958185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5958436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5958870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5959304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5959539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5959771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5959865Z ok (4.456s) 2022-11-23T02:00:45.5959872Z 2022-11-23T02:00:45.5960216Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5960324Z Ran 1 test in 4.457s 2022-11-23T02:00:45.5960330Z 2022-11-23T02:00:45.5960420Z OK 2022-11-23T02:00:45.5960426Z 2022-11-23T02:00:45.5960534Z Generating XML reports... 2022-11-23T02:00:45.5961029Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014550.xml 2022-11-23T02:00:45.5961370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5961773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5961951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5962358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5962549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5962555Z 2022-11-23T02:00:45.5962655Z Running tests... 2022-11-23T02:00:45.5962940Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5963307Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12719 2022-11-23T02:00:45.5963586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12720 2022-11-23T02:00:45.5963863Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5964264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5964438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5964851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5965042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5965289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5965693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5965867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5966327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5966522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5966776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5967215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5967637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5967871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5968112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5968211Z ok (4.455s) 2022-11-23T02:00:45.5968219Z 2022-11-23T02:00:45.5968503Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5968613Z Ran 1 test in 4.455s 2022-11-23T02:00:45.5968623Z 2022-11-23T02:00:45.5968709Z OK 2022-11-23T02:00:45.5968716Z 2022-11-23T02:00:45.5968835Z Generating XML reports... 2022-11-23T02:00:45.5969328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014558.xml 2022-11-23T02:00:45.5969668Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5970069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5970243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5970660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5970853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5970860Z 2022-11-23T02:00:45.5970962Z Running tests... 2022-11-23T02:00:45.5971252Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5971568Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:00:45.5971575Z 2022-11-23T02:00:45.5971858Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5971966Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5971973Z 2022-11-23T02:00:45.5972074Z OK (skipped=1) 2022-11-23T02:00:45.5972081Z 2022-11-23T02:00:45.5972200Z Generating XML reports... 2022-11-23T02:00:45.5972688Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014605.xml 2022-11-23T02:00:45.5973077Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5973462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5973642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5974053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5974248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5974255Z 2022-11-23T02:00:45.5974354Z Running tests... 2022-11-23T02:00:45.5974637Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5975007Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12967 2022-11-23T02:00:45.5975234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12968 2022-11-23T02:00:45.5975509Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5975949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5976123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5976530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5976723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5976974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5977368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5977547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5977959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5978152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5978406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5978842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5979273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.5979512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.5979762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.5979985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.5980229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.5980664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.5981098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.5981193Z ok (4.373s) 2022-11-23T02:00:45.5981200Z 2022-11-23T02:00:45.5981483Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5981588Z Ran 1 test in 4.373s 2022-11-23T02:00:45.5981595Z 2022-11-23T02:00:45.5981681Z OK 2022-11-23T02:00:45.5981687Z 2022-11-23T02:00:45.5981805Z Generating XML reports... 2022-11-23T02:00:45.5982292Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014608.xml 2022-11-23T02:00:45.5982683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5983080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5983261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5983675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5983870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5983877Z 2022-11-23T02:00:45.5983982Z Running tests... 2022-11-23T02:00:45.5984267Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5984578Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:00:45.5984589Z 2022-11-23T02:00:45.5984866Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5984972Z Ran 1 test in 0.002s 2022-11-23T02:00:45.5984978Z 2022-11-23T02:00:45.5985080Z OK (skipped=1) 2022-11-23T02:00:45.5985086Z 2022-11-23T02:00:45.5985204Z Generating XML reports... 2022-11-23T02:00:45.5985740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014616.xml 2022-11-23T02:00:45.5986064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.5986462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5986638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5987052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5987250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5987257Z 2022-11-23T02:00:45.5987355Z Running tests... 2022-11-23T02:00:45.5987638Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.5988003Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13217 2022-11-23T02:00:45.5988223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13218 2022-11-23T02:00:45.5988497Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.5988890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5989062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5989471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5989665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5989919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.5990321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.5990500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.5990908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.5991104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.5991357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.5991780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6014443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6014883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6015232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6015388Z skip: Skipped due to small world size. (3.875s) 2022-11-23T02:00:45.6015398Z 2022-11-23T02:00:45.6015752Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6015850Z Ran 1 test in 3.876s 2022-11-23T02:00:45.6015858Z 2022-11-23T02:00:45.6015950Z OK (skipped=1) 2022-11-23T02:00:45.6015956Z 2022-11-23T02:00:45.6016066Z Generating XML reports... 2022-11-23T02:00:45.6016569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014619.xml 2022-11-23T02:00:45.6016907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6017303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6017472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6018085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6018272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6018280Z 2022-11-23T02:00:45.6018372Z Running tests... 2022-11-23T02:00:45.6018655Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6018988Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13396 2022-11-23T02:00:45.6019202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13397 2022-11-23T02:00:45.6019473Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6019869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6020043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6020447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6020631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6020876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6021266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6021433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6021835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6022014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6022258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6022687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6023277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6023503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6023725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6023961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.6024297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.6024719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6025143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6025241Z ok (4.689s) 2022-11-23T02:00:45.6025249Z 2022-11-23T02:00:45.6025539Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6025645Z Ran 1 test in 4.689s 2022-11-23T02:00:45.6025652Z 2022-11-23T02:00:45.6025737Z OK 2022-11-23T02:00:45.6025744Z 2022-11-23T02:00:45.6025848Z Generating XML reports... 2022-11-23T02:00:45.6026346Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014626.xml 2022-11-23T02:00:45.6026692Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6027092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6027268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6027729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6027927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6027935Z 2022-11-23T02:00:45.6028037Z Running tests... 2022-11-23T02:00:45.6028327Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6028796Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:00:45.6028803Z 2022-11-23T02:00:45.6029086Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6029191Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6029198Z 2022-11-23T02:00:45.6029298Z OK (skipped=1) 2022-11-23T02:00:45.6029304Z 2022-11-23T02:00:45.6029418Z Generating XML reports... 2022-11-23T02:00:45.6029913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014634.xml 2022-11-23T02:00:45.6030255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6030659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6030835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6031249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6031444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6031455Z 2022-11-23T02:00:45.6031556Z Running tests... 2022-11-23T02:00:45.6031836Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6032276Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s) 2022-11-23T02:00:45.6032298Z 2022-11-23T02:00:45.6032571Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6032679Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6032686Z 2022-11-23T02:00:45.6032787Z OK (skipped=1) 2022-11-23T02:00:45.6032793Z 2022-11-23T02:00:45.6032911Z Generating XML reports... 2022-11-23T02:00:45.6033410Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014637.xml 2022-11-23T02:00:45.6033750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6034153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6034375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6034786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6034979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6034986Z 2022-11-23T02:00:45.6035086Z Running tests... 2022-11-23T02:00:45.6035368Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6035630Z test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s) 2022-11-23T02:00:45.6035638Z 2022-11-23T02:00:45.6035919Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6036027Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6036034Z 2022-11-23T02:00:45.6036135Z OK (skipped=1) 2022-11-23T02:00:45.6036142Z 2022-11-23T02:00:45.6036263Z Generating XML reports... 2022-11-23T02:00:45.6036762Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014640.xml 2022-11-23T02:00:45.6037105Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6037548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6037727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6038142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6038319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6038339Z 2022-11-23T02:00:45.6038428Z Running tests... 2022-11-23T02:00:45.6038717Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6039045Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13769 2022-11-23T02:00:45.6039272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13770 2022-11-23T02:00:45.6039555Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6039953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6040201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6040614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6040804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6041051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6041449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6041620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6042025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6042219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6042471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6042908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6043340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6043579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6043873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6044019Z ok (5.162s) 2022-11-23T02:00:45.6044027Z 2022-11-23T02:00:45.6044321Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6044412Z Ran 1 test in 5.163s 2022-11-23T02:00:45.6044436Z 2022-11-23T02:00:45.6044510Z OK 2022-11-23T02:00:45.6044517Z 2022-11-23T02:00:45.6044637Z Generating XML reports... 2022-11-23T02:00:45.6045131Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014643.xml 2022-11-23T02:00:45.6045471Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6045873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6046046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6046463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6046651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6046658Z 2022-11-23T02:00:45.6046760Z Running tests... 2022-11-23T02:00:45.6047092Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6047370Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.001s) 2022-11-23T02:00:45.6047377Z 2022-11-23T02:00:45.6047660Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6047765Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6047771Z 2022-11-23T02:00:45.6047869Z OK (skipped=1) 2022-11-23T02:00:45.6047876Z 2022-11-23T02:00:45.6047995Z Generating XML reports... 2022-11-23T02:00:45.6048492Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014651.xml 2022-11-23T02:00:45.6048833Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6049232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6049407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6049820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6050015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6050022Z 2022-11-23T02:00:45.6050125Z Running tests... 2022-11-23T02:00:45.6050396Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6050736Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14013 2022-11-23T02:00:45.6050962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14014 2022-11-23T02:00:45.6051246Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6051645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6051827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6052239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6052430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6052681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6053076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6053243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6053703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6053887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6054139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6054581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6055017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6055251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6055485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6055646Z skip: Skipped due to small world size. (3.955s) 2022-11-23T02:00:45.6055657Z 2022-11-23T02:00:45.6055945Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6056055Z Ran 1 test in 3.956s 2022-11-23T02:00:45.6056062Z 2022-11-23T02:00:45.6056168Z OK (skipped=1) 2022-11-23T02:00:45.6056174Z 2022-11-23T02:00:45.6056293Z Generating XML reports... 2022-11-23T02:00:45.6056822Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014654.xml 2022-11-23T02:00:45.6057163Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6057564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6057741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6058154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6058353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6058360Z 2022-11-23T02:00:45.6058461Z Running tests... 2022-11-23T02:00:45.6058744Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6059014Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s) 2022-11-23T02:00:45.6059021Z 2022-11-23T02:00:45.6059306Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6059414Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6059421Z 2022-11-23T02:00:45.6059522Z OK (skipped=1) 2022-11-23T02:00:45.6059529Z 2022-11-23T02:00:45.6059646Z Generating XML reports... 2022-11-23T02:00:45.6060139Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014701.xml 2022-11-23T02:00:45.6060475Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6060875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6061046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6061450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6061637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6061644Z 2022-11-23T02:00:45.6061744Z Running tests... 2022-11-23T02:00:45.6062027Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6062363Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14250 2022-11-23T02:00:45.6062573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14251 2022-11-23T02:00:45.6062856Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6063302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6063479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6063891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6064081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6064336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6064730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6064903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6065309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6065503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6065753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6066232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6066668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6066902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6067136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6067298Z skip: Skipped due to small world size. (3.853s) 2022-11-23T02:00:45.6067306Z 2022-11-23T02:00:45.6067592Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6067703Z Ran 1 test in 3.853s 2022-11-23T02:00:45.6067711Z 2022-11-23T02:00:45.6067811Z OK (skipped=1) 2022-11-23T02:00:45.6067817Z 2022-11-23T02:00:45.6067934Z Generating XML reports... 2022-11-23T02:00:45.6068432Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014704.xml 2022-11-23T02:00:45.6068771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6069155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6069333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6069747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6069941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6069951Z 2022-11-23T02:00:45.6070053Z Running tests... 2022-11-23T02:00:45.6070334Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6070624Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.001s) 2022-11-23T02:00:45.6070631Z 2022-11-23T02:00:45.6070920Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6071029Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6071035Z 2022-11-23T02:00:45.6071136Z OK (skipped=1) 2022-11-23T02:00:45.6071142Z 2022-11-23T02:00:45.6071260Z Generating XML reports... 2022-11-23T02:00:45.6071755Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014711.xml 2022-11-23T02:00:45.6072091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6072486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6072708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6073115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6073309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6073316Z 2022-11-23T02:00:45.6073417Z Running tests... 2022-11-23T02:00:45.6073702Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6073986Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s) 2022-11-23T02:00:45.6073993Z 2022-11-23T02:00:45.6074275Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6074381Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6074388Z 2022-11-23T02:00:45.6074483Z OK (skipped=1) 2022-11-23T02:00:45.6074490Z 2022-11-23T02:00:45.6074592Z Generating XML reports... 2022-11-23T02:00:45.6075089Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014714.xml 2022-11-23T02:00:45.6075428Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6075868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6076046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6076459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6076651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6076658Z 2022-11-23T02:00:45.6076764Z Running tests... 2022-11-23T02:00:45.6077046Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6077328Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.001s) 2022-11-23T02:00:45.6077338Z 2022-11-23T02:00:45.6077616Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6077721Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6077728Z 2022-11-23T02:00:45.6077827Z OK (skipped=1) 2022-11-23T02:00:45.6077833Z 2022-11-23T02:00:45.6077953Z Generating XML reports... 2022-11-23T02:00:45.6078443Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014717.xml 2022-11-23T02:00:45.6078784Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6079181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6079352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6079758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6079951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6079958Z 2022-11-23T02:00:45.6080059Z Running tests... 2022-11-23T02:00:45.6080404Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6080660Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s) 2022-11-23T02:00:45.6080684Z 2022-11-23T02:00:45.6080957Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6081066Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6081072Z 2022-11-23T02:00:45.6081174Z OK (skipped=1) 2022-11-23T02:00:45.6081180Z 2022-11-23T02:00:45.6081296Z Generating XML reports... 2022-11-23T02:00:45.6081788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014720.xml 2022-11-23T02:00:45.6082127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6082590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6082763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6083182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6083378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6083385Z 2022-11-23T02:00:45.6083485Z Running tests... 2022-11-23T02:00:45.6083769Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6084044Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s) 2022-11-23T02:00:45.6084051Z 2022-11-23T02:00:45.6084334Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6084444Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6084451Z 2022-11-23T02:00:45.6084549Z OK (skipped=1) 2022-11-23T02:00:45.6084555Z 2022-11-23T02:00:45.6084673Z Generating XML reports... 2022-11-23T02:00:45.6085165Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014724.xml 2022-11-23T02:00:45.6085550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6085951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6086127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6086537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6086714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6086737Z 2022-11-23T02:00:45.6086823Z Running tests... 2022-11-23T02:00:45.6087111Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6087471Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14719 2022-11-23T02:00:45.6087700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14720 2022-11-23T02:00:45.6087981Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6088382Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6088558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6088969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6089163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6089418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6089814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6089989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6090397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6090585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6090836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6091270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6091702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6092003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6092233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6092480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.6092726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.6093159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6093403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:00:45.6093819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6094069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:00:45.6094502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:00:45.6094973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:00:45.6095073Z ok (3.875s) 2022-11-23T02:00:45.6095080Z 2022-11-23T02:00:45.6095369Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6095478Z Ran 1 test in 3.876s 2022-11-23T02:00:45.6095485Z 2022-11-23T02:00:45.6095571Z OK 2022-11-23T02:00:45.6095578Z 2022-11-23T02:00:45.6095699Z Generating XML reports... 2022-11-23T02:00:45.6096195Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014727.xml 2022-11-23T02:00:45.6096536Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6096940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6097115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6097531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6097726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6097733Z 2022-11-23T02:00:45.6097836Z Running tests... 2022-11-23T02:00:45.6098121Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6098466Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14906 2022-11-23T02:00:45.6098689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14907 2022-11-23T02:00:45.6098967Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6099369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6099550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6099963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6100145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6100399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6100803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6100979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6101390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6101633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6101884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6102325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6102754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6102989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6103220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6103316Z ok (4.457s) 2022-11-23T02:00:45.6103323Z 2022-11-23T02:00:45.6103612Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6103722Z Ran 1 test in 4.458s 2022-11-23T02:00:45.6103729Z 2022-11-23T02:00:45.6103815Z OK 2022-11-23T02:00:45.6103821Z 2022-11-23T02:00:45.6103936Z Generating XML reports... 2022-11-23T02:00:45.6104435Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014734.xml 2022-11-23T02:00:45.6104818Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6105221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6105398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6105811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6106003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6106010Z 2022-11-23T02:00:45.6106101Z Running tests... 2022-11-23T02:00:45.6106388Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6106744Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15096 2022-11-23T02:00:45.6106973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15097 2022-11-23T02:00:45.6107250Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6107649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6107826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6108234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6108426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6108683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6109077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6109255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6109660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6109847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6110098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6110530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6110959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6111240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6111470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6111625Z skip: Skipped due to small world size. (3.859s) 2022-11-23T02:00:45.6111635Z 2022-11-23T02:00:45.6111925Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6112030Z Ran 1 test in 3.860s 2022-11-23T02:00:45.6112037Z 2022-11-23T02:00:45.6112136Z OK (skipped=1) 2022-11-23T02:00:45.6112143Z 2022-11-23T02:00:45.6112246Z Generating XML reports... 2022-11-23T02:00:45.6112742Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014741.xml 2022-11-23T02:00:45.6113084Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6113486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6113666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6114077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6114315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6114323Z 2022-11-23T02:00:45.6114428Z Running tests... 2022-11-23T02:00:45.6114714Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6115063Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15275 2022-11-23T02:00:45.6115289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15276 2022-11-23T02:00:45.6115570Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6115972Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6116149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6116564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6116755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6117007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6117408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6117584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6117994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6118190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6118439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6118864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6119295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6119530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6119763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6119863Z ok (4.255s) 2022-11-23T02:00:45.6119870Z 2022-11-23T02:00:45.6120220Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6120327Z Ran 1 test in 4.255s 2022-11-23T02:00:45.6120392Z 2022-11-23T02:00:45.6120477Z OK 2022-11-23T02:00:45.6120483Z 2022-11-23T02:00:45.6120601Z Generating XML reports... 2022-11-23T02:00:45.6121100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014748.xml 2022-11-23T02:00:45.6121443Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6121844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6122019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6122430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6122622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6122629Z 2022-11-23T02:00:45.6122730Z Running tests... 2022-11-23T02:00:45.6124630Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6124985Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15454 2022-11-23T02:00:45.6125209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15455 2022-11-23T02:00:45.6125560Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6125966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6126143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6126555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6126751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6126995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6127392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6127565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6127978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6128168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6128418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6128851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6129278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6129513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6129747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6129841Z ok (3.865s) 2022-11-23T02:00:45.6129849Z 2022-11-23T02:00:45.6130135Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6130239Z Ran 1 test in 3.866s 2022-11-23T02:00:45.6130246Z 2022-11-23T02:00:45.6130327Z OK 2022-11-23T02:00:45.6130334Z 2022-11-23T02:00:45.6130453Z Generating XML reports... 2022-11-23T02:00:45.6130949Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014756.xml 2022-11-23T02:00:45.6131292Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6131697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6131923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6132337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6132531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6132538Z 2022-11-23T02:00:45.6132643Z Running tests... 2022-11-23T02:00:45.6132918Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6133281Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15633 2022-11-23T02:00:45.6133506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15634 2022-11-23T02:00:45.6133781Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6134178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6134360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6134767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6135006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6135263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6135669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6135845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6136256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6136453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6136704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6137138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6137573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6137806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6138043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6138141Z ok (4.566s) 2022-11-23T02:00:45.6138148Z 2022-11-23T02:00:45.6138437Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6138546Z Ran 1 test in 4.566s 2022-11-23T02:00:45.6138552Z 2022-11-23T02:00:45.6138640Z OK 2022-11-23T02:00:45.6138647Z 2022-11-23T02:00:45.6138769Z Generating XML reports... 2022-11-23T02:00:45.6139248Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014802.xml 2022-11-23T02:00:45.6139590Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6139990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6140166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6140573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6140767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6140774Z 2022-11-23T02:00:45.6140873Z Running tests... 2022-11-23T02:00:45.6141163Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6141508Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15819 2022-11-23T02:00:45.6141777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15820 2022-11-23T02:00:45.6142058Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6142460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6142639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6143048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6143239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6143492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6143895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6144074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6144534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6144728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6144979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6145415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6145828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6146064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6146302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6146399Z ok (4.660s) 2022-11-23T02:00:45.6146407Z 2022-11-23T02:00:45.6146694Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6146798Z Ran 1 test in 4.660s 2022-11-23T02:00:45.6146808Z 2022-11-23T02:00:45.6146895Z OK 2022-11-23T02:00:45.6146902Z 2022-11-23T02:00:45.6147020Z Generating XML reports... 2022-11-23T02:00:45.6147509Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014810.xml 2022-11-23T02:00:45.6147846Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6148240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6148414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6148822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6149009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6149016Z 2022-11-23T02:00:45.6149118Z Running tests... 2022-11-23T02:00:45.6149408Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6149757Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16005 2022-11-23T02:00:45.6149981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16006 2022-11-23T02:00:45.6150265Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6150660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6150883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6151298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6151477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6151732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6152170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6152565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6152743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6153152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6153349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6153601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6154034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6154314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6154545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6154636Z ok (3.751s) 2022-11-23T02:00:45.6154644Z 2022-11-23T02:00:45.6154928Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6155032Z Ran 1 test in 3.752s 2022-11-23T02:00:45.6155040Z 2022-11-23T02:00:45.6155121Z OK 2022-11-23T02:00:45.6155127Z 2022-11-23T02:00:45.6155246Z Generating XML reports... 2022-11-23T02:00:45.6155744Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014818.xml 2022-11-23T02:00:45.6156088Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6156493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6156676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6157089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6157286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6157292Z 2022-11-23T02:00:45.6157395Z Running tests... 2022-11-23T02:00:45.6157669Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6157937Z test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.6157947Z 2022-11-23T02:00:45.6158232Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6158337Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6158344Z 2022-11-23T02:00:45.6158444Z OK (skipped=1) 2022-11-23T02:00:45.6158450Z 2022-11-23T02:00:45.6158570Z Generating XML reports... 2022-11-23T02:00:45.6159066Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014825.xml 2022-11-23T02:00:45.6159405Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6159802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6159976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6160476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6160724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6160731Z 2022-11-23T02:00:45.6160830Z Running tests... 2022-11-23T02:00:45.6161114Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6162117Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.537s) 2022-11-23T02:00:45.6162125Z 2022-11-23T02:00:45.6162407Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6162511Z Ran 1 test in 0.537s 2022-11-23T02:00:45.6162518Z 2022-11-23T02:00:45.6162621Z OK (skipped=1) 2022-11-23T02:00:45.6162628Z 2022-11-23T02:00:45.6162744Z Generating XML reports... 2022-11-23T02:00:45.6163241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014828.xml 2022-11-23T02:00:45.6163587Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6164064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6164243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6164657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6164848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6164855Z 2022-11-23T02:00:45.6164944Z Running tests... 2022-11-23T02:00:45.6165227Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6165507Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.6165518Z 2022-11-23T02:00:45.6165802Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6165907Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6165913Z 2022-11-23T02:00:45.6166011Z OK (skipped=1) 2022-11-23T02:00:45.6166018Z 2022-11-23T02:00:45.6166132Z Generating XML reports... 2022-11-23T02:00:45.6166622Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014831.xml 2022-11-23T02:00:45.6166957Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6167359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6167535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6167947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6168138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6168145Z 2022-11-23T02:00:45.6168245Z Running tests... 2022-11-23T02:00:45.6168529Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6168807Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.6168814Z 2022-11-23T02:00:45.6169096Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6169201Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6169207Z 2022-11-23T02:00:45.6169307Z OK (skipped=1) 2022-11-23T02:00:45.6169314Z 2022-11-23T02:00:45.6169426Z Generating XML reports... 2022-11-23T02:00:45.6169922Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014834.xml 2022-11-23T02:00:45.6170262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6170709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6170871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6171286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6171484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6171491Z 2022-11-23T02:00:45.6171590Z Running tests... 2022-11-23T02:00:45.6171877Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6172155Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.001s) 2022-11-23T02:00:45.6172162Z 2022-11-23T02:00:45.6172441Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6172546Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6172552Z 2022-11-23T02:00:45.6172653Z OK (skipped=1) 2022-11-23T02:00:45.6172659Z 2022-11-23T02:00:45.6172776Z Generating XML reports... 2022-11-23T02:00:45.6173258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014837.xml 2022-11-23T02:00:45.6173634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6174037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6174213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6174626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6174818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6174826Z 2022-11-23T02:00:45.6174927Z Running tests... 2022-11-23T02:00:45.6175213Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6176215Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.614s) 2022-11-23T02:00:45.6176224Z 2022-11-23T02:00:45.6176507Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6176612Z Ran 1 test in 0.615s 2022-11-23T02:00:45.6176619Z 2022-11-23T02:00:45.6176719Z OK (skipped=1) 2022-11-23T02:00:45.6176725Z 2022-11-23T02:00:45.6176844Z Generating XML reports... 2022-11-23T02:00:45.6177337Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014841.xml 2022-11-23T02:00:45.6177677Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6178066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6178245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6178664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6178856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6178862Z 2022-11-23T02:00:45.6178962Z Running tests... 2022-11-23T02:00:45.6179246Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6179807Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:00:45.6179815Z 2022-11-23T02:00:45.6180094Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6180245Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6180253Z 2022-11-23T02:00:45.6180348Z OK (skipped=1) 2022-11-23T02:00:45.6180354Z 2022-11-23T02:00:45.6180473Z Generating XML reports... 2022-11-23T02:00:45.6180975Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014844.xml 2022-11-23T02:00:45.6181316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6181718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6181896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6182308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6182504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6182512Z 2022-11-23T02:00:45.6182619Z Running tests... 2022-11-23T02:00:45.6182906Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6183476Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:00:45.6183532Z 2022-11-23T02:00:45.6183820Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6183927Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6183933Z 2022-11-23T02:00:45.6184032Z OK (skipped=1) 2022-11-23T02:00:45.6184038Z 2022-11-23T02:00:45.6184157Z Generating XML reports... 2022-11-23T02:00:45.6184637Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014847.xml 2022-11-23T02:00:45.6184977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6185380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6185563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6185970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6186166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6186173Z 2022-11-23T02:00:45.6186270Z Running tests... 2022-11-23T02:00:45.6186551Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6186892Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16648 2022-11-23T02:00:45.6187111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16649 2022-11-23T02:00:45.6187391Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6187796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6187975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6188391Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6188584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6188841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6189239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6189417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6189825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6190062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6190322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6190774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6191207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6191425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6191662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6191931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocy3f33a 2022-11-23T02:00:45.6192209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocy3f33a/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6192471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobqtctn5 2022-11-23T02:00:45.6192740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobqtctn5/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6193026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6193267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6193365Z ok (6.676s) 2022-11-23T02:00:45.6193373Z 2022-11-23T02:00:45.6193665Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6193773Z Ran 1 test in 6.676s 2022-11-23T02:00:45.6193780Z 2022-11-23T02:00:45.6193871Z OK 2022-11-23T02:00:45.6193877Z 2022-11-23T02:00:45.6193998Z Generating XML reports... 2022-11-23T02:00:45.6194490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014850.xml 2022-11-23T02:00:45.6194836Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6195239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6195414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6195826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6196021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6196028Z 2022-11-23T02:00:45.6196130Z Running tests... 2022-11-23T02:00:45.6196419Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6196768Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16844 2022-11-23T02:00:45.6196978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16845 2022-11-23T02:00:45.6197261Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6197662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6197841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6198247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6198435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6198688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6199086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6199258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6199717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6199908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6200246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6200685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6201119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6201354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6201592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6201861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlx0cl_z 2022-11-23T02:00:45.6202147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlx0cl_z/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6202406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_o7optz 2022-11-23T02:00:45.6202738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_o7optz/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6202986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6203230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6203472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6203696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6203794Z ok (6.379s) 2022-11-23T02:00:45.6203802Z 2022-11-23T02:00:45.6204096Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6204206Z Ran 1 test in 6.380s 2022-11-23T02:00:45.6204213Z 2022-11-23T02:00:45.6204296Z OK 2022-11-23T02:00:45.6204303Z 2022-11-23T02:00:45.6204422Z Generating XML reports... 2022-11-23T02:00:45.6204922Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014900.xml 2022-11-23T02:00:45.6205256Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6205657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6205836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6206245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6206439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6206449Z 2022-11-23T02:00:45.6206552Z Running tests... 2022-11-23T02:00:45.6206841Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6207854Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.618s) 2022-11-23T02:00:45.6207863Z 2022-11-23T02:00:45.6208152Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6208260Z Ran 1 test in 0.618s 2022-11-23T02:00:45.6208267Z 2022-11-23T02:00:45.6208368Z OK (skipped=1) 2022-11-23T02:00:45.6208375Z 2022-11-23T02:00:45.6208492Z Generating XML reports... 2022-11-23T02:00:45.6208983Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014910.xml 2022-11-23T02:00:45.6209382Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6209781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6209958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6210363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6210541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6210562Z 2022-11-23T02:00:45.6210650Z Running tests... 2022-11-23T02:00:45.6210933Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6211967Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.619s) 2022-11-23T02:00:45.6211990Z 2022-11-23T02:00:45.6212257Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6212437Z Ran 1 test in 0.619s 2022-11-23T02:00:45.6212444Z 2022-11-23T02:00:45.6212546Z OK (skipped=1) 2022-11-23T02:00:45.6212553Z 2022-11-23T02:00:45.6212673Z Generating XML reports... 2022-11-23T02:00:45.6213173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014913.xml 2022-11-23T02:00:45.6213512Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6213917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6214098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6214517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6214713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6214720Z 2022-11-23T02:00:45.6214826Z Running tests... 2022-11-23T02:00:45.6215113Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6215622Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.003s) 2022-11-23T02:00:45.6215629Z 2022-11-23T02:00:45.6215911Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6216019Z Ran 1 test in 0.003s 2022-11-23T02:00:45.6216026Z 2022-11-23T02:00:45.6216126Z OK (skipped=1) 2022-11-23T02:00:45.6216132Z 2022-11-23T02:00:45.6216248Z Generating XML reports... 2022-11-23T02:00:45.6216743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014917.xml 2022-11-23T02:00:45.6217079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6217480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6217652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6218059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6218239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6218259Z 2022-11-23T02:00:45.6218345Z Running tests... 2022-11-23T02:00:45.6218629Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6219014Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17214 2022-11-23T02:00:45.6219285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17215 2022-11-23T02:00:45.6219565Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6219970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6220148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6220554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6220745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6221000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6221401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6221576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6221983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6222220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6222480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6222922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6223352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6223585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6223822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6224091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkseuyxgw 2022-11-23T02:00:45.6224371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkseuyxgw/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6224633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78xblhfh 2022-11-23T02:00:45.6224904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78xblhfh/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6224985Z ok (4.490s) 2022-11-23T02:00:45.6224992Z 2022-11-23T02:00:45.6225280Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6225386Z Ran 1 test in 4.491s 2022-11-23T02:00:45.6225393Z 2022-11-23T02:00:45.6225480Z OK 2022-11-23T02:00:45.6225487Z 2022-11-23T02:00:45.6225608Z Generating XML reports... 2022-11-23T02:00:45.6226268Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014920.xml 2022-11-23T02:00:45.6226611Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6227017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6227198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6227610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6227801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6227808Z 2022-11-23T02:00:45.6227910Z Running tests... 2022-11-23T02:00:45.6228197Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6228530Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17400 2022-11-23T02:00:45.6228808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17401 2022-11-23T02:00:45.6229086Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6229493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6229668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6230076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6230272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6230528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6230935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6231099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6231510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6231864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6232046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6232355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6232664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6232834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6233005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6233200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyuxwxcap 2022-11-23T02:00:45.6233406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyuxwxcap/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6233598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lfthr9x 2022-11-23T02:00:45.6233798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lfthr9x/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6233969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6234142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6234212Z ok (6.568s) 2022-11-23T02:00:45.6234217Z 2022-11-23T02:00:45.6234422Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6234498Z Ran 1 test in 6.568s 2022-11-23T02:00:45.6234505Z 2022-11-23T02:00:45.6234571Z OK 2022-11-23T02:00:45.6234576Z 2022-11-23T02:00:45.6234662Z Generating XML reports... 2022-11-23T02:00:45.6235015Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014928.xml 2022-11-23T02:00:45.6235258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6235542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6235657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6235953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6236089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6236093Z 2022-11-23T02:00:45.6236166Z Running tests... 2022-11-23T02:00:45.6236372Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6236799Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.004s) 2022-11-23T02:00:45.6236804Z 2022-11-23T02:00:45.6237008Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6237092Z Ran 1 test in 0.004s 2022-11-23T02:00:45.6237097Z 2022-11-23T02:00:45.6237173Z OK (skipped=1) 2022-11-23T02:00:45.6237178Z 2022-11-23T02:00:45.6237267Z Generating XML reports... 2022-11-23T02:00:45.6237620Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014937.xml 2022-11-23T02:00:45.6237861Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6238148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6238275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6238568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6238709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6238713Z 2022-11-23T02:00:45.6238825Z Running tests... 2022-11-23T02:00:45.6239030Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6239388Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s) 2022-11-23T02:00:45.6239393Z 2022-11-23T02:00:45.6239595Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6239676Z Ran 1 test in 0.003s 2022-11-23T02:00:45.6239681Z 2022-11-23T02:00:45.6239754Z OK (skipped=1) 2022-11-23T02:00:45.6239758Z 2022-11-23T02:00:45.6239844Z Generating XML reports... 2022-11-23T02:00:45.6240236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014941.xml 2022-11-23T02:00:45.6240821Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6241136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6241279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6241595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6241744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6241750Z 2022-11-23T02:00:45.6241834Z Running tests... 2022-11-23T02:00:45.6242054Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6242330Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s) 2022-11-23T02:00:45.6242338Z 2022-11-23T02:00:45.6242563Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6242650Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6242655Z 2022-11-23T02:00:45.6242736Z OK (skipped=1) 2022-11-23T02:00:45.6242741Z 2022-11-23T02:00:45.6242841Z Generating XML reports... 2022-11-23T02:00:45.6243223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014944.xml 2022-11-23T02:00:45.6243490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6243801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6243945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6244261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6244458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6244464Z 2022-11-23T02:00:45.6244544Z Running tests... 2022-11-23T02:00:45.6244771Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6245120Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.004s) 2022-11-23T02:00:45.6245126Z 2022-11-23T02:00:45.6245342Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6245412Z Ran 1 test in 0.004s 2022-11-23T02:00:45.6245427Z 2022-11-23T02:00:45.6245492Z OK (skipped=1) 2022-11-23T02:00:45.6245497Z 2022-11-23T02:00:45.6245589Z Generating XML reports... 2022-11-23T02:00:45.6245962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014947.xml 2022-11-23T02:00:45.6246218Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6246530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6246670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6247027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6247180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6247185Z 2022-11-23T02:00:45.6247270Z Running tests... 2022-11-23T02:00:45.6247496Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6247878Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:00:45.6247884Z 2022-11-23T02:00:45.6248102Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6248193Z Ran 1 test in 0.003s 2022-11-23T02:00:45.6248198Z 2022-11-23T02:00:45.6248279Z OK (skipped=1) 2022-11-23T02:00:45.6248284Z 2022-11-23T02:00:45.6248380Z Generating XML reports... 2022-11-23T02:00:45.6248761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014950.xml 2022-11-23T02:00:45.6249025Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6249334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6249472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6249790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6249938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6249944Z 2022-11-23T02:00:45.6250026Z Running tests... 2022-11-23T02:00:45.6250233Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6251003Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.635s) 2022-11-23T02:00:45.6251022Z 2022-11-23T02:00:45.6251353Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6251421Z Ran 1 test in 0.635s 2022-11-23T02:00:45.6251439Z 2022-11-23T02:00:45.6251503Z OK (skipped=1) 2022-11-23T02:00:45.6251508Z 2022-11-23T02:00:45.6251600Z Generating XML reports... 2022-11-23T02:00:45.6251968Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014953.xml 2022-11-23T02:00:45.6252262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6252669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6252801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6253099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6253240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6253245Z 2022-11-23T02:00:45.6253323Z Running tests... 2022-11-23T02:00:45.6253527Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6254238Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.500s) 2022-11-23T02:00:45.6254246Z 2022-11-23T02:00:45.6254450Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6254533Z Ran 1 test in 0.500s 2022-11-23T02:00:45.6254538Z 2022-11-23T02:00:45.6254652Z OK (skipped=1) 2022-11-23T02:00:45.6254657Z 2022-11-23T02:00:45.6254745Z Generating XML reports... 2022-11-23T02:00:45.6255099Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014957.xml 2022-11-23T02:00:45.6255342Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6255627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6255753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6256049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6256190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6256195Z 2022-11-23T02:00:45.6256266Z Running tests... 2022-11-23T02:00:45.6256471Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6256734Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18002 2022-11-23T02:00:45.6256898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18003 2022-11-23T02:00:45.6257085Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6257369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6257500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6257793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6257933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6258120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6258408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6258539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6258836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6258974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6259159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6259507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6259816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6259986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6260154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.6260325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6260494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.6260797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6261098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6261289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmfr3r2i 2022-11-23T02:00:45.6261522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmfr3r2i/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6261705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbrco_a0 2022-11-23T02:00:45.6261903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbrco_a0/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6262077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6262249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6262416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6262584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6262653Z ok (7.063s) 2022-11-23T02:00:45.6262658Z 2022-11-23T02:00:45.6262865Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6262940Z Ran 1 test in 7.064s 2022-11-23T02:00:45.6262945Z 2022-11-23T02:00:45.6263007Z OK 2022-11-23T02:00:45.6263014Z 2022-11-23T02:00:45.6263100Z Generating XML reports... 2022-11-23T02:00:45.6263451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015000.xml 2022-11-23T02:00:45.6263694Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6263977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6264102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6264381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6264520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6264538Z 2022-11-23T02:00:45.6264598Z Running tests... 2022-11-23T02:00:45.6264806Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6265054Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18200 2022-11-23T02:00:45.6265212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18201 2022-11-23T02:00:45.6265408Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6265689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6265815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6266145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6266285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6266469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6266752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6266875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6267168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6267305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6267485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6267798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6268109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6268321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6268527Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:00:45.6268693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6268891Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:00:45.6269077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8f5s_u62 2022-11-23T02:00:45.6269262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8f5s_u62/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6269452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2g_fpat 2022-11-23T02:00:45.6269649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2g_fpat/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6269826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6269998Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6270170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6270341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6270550Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:00:45.6270750Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:00:45.6270955Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:00:45.6271152Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:00:45.6271327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6271499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6271674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6271847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6272054Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:00:45.6272256Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:00:45.6272496Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:00:45.6272694Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:00:45.6272867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6273034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6273204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6273371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6273429Z ok (7.177s) 2022-11-23T02:00:45.6273434Z 2022-11-23T02:00:45.6273653Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6273731Z Ran 1 test in 7.178s 2022-11-23T02:00:45.6273738Z 2022-11-23T02:00:45.6273800Z OK 2022-11-23T02:00:45.6273806Z 2022-11-23T02:00:45.6273890Z Generating XML reports... 2022-11-23T02:00:45.6274245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015010.xml 2022-11-23T02:00:45.6274534Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6274820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6274943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6275233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6275371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6275376Z 2022-11-23T02:00:45.6275449Z Running tests... 2022-11-23T02:00:45.6275659Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6276370Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.622s) 2022-11-23T02:00:45.6276376Z 2022-11-23T02:00:45.6276581Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6276659Z Ran 1 test in 0.622s 2022-11-23T02:00:45.6276664Z 2022-11-23T02:00:45.6276741Z OK (skipped=1) 2022-11-23T02:00:45.6276746Z 2022-11-23T02:00:45.6276839Z Generating XML reports... 2022-11-23T02:00:45.6277193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015020.xml 2022-11-23T02:00:45.6277438Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6277728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6277857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6278154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6278301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6278306Z 2022-11-23T02:00:45.6278383Z Running tests... 2022-11-23T02:00:45.6278576Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6278823Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18454 2022-11-23T02:00:45.6278985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18455 2022-11-23T02:00:45.6279224Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6279506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6279630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6279919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6280054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6280287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6280757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6280886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6281194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6281342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6281531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6281908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6282235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6282411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6282849Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:00:45.6283027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6283464Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:00:45.6283670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9qc_ksm 2022-11-23T02:00:45.6283879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9qc_ksm/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6284073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptb4ivm0j 2022-11-23T02:00:45.6284283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptb4ivm0j/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6284465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6284651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6284868Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:00:45.6285075Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:00:45.6285294Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:00:45.6285522Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:00:45.6285775Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:00:45.6286071Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:00:45.6286323Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:00:45.6286573Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:00:45.6286750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6286929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6287002Z ok (7.369s) 2022-11-23T02:00:45.6287007Z 2022-11-23T02:00:45.6287227Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6287312Z Ran 1 test in 7.369s 2022-11-23T02:00:45.6287318Z 2022-11-23T02:00:45.6287382Z OK 2022-11-23T02:00:45.6287387Z 2022-11-23T02:00:45.6287482Z Generating XML reports... 2022-11-23T02:00:45.6287854Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015024.xml 2022-11-23T02:00:45.6288160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6288468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6288605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6288916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6289064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6289069Z 2022-11-23T02:00:45.6289146Z Running tests... 2022-11-23T02:00:45.6289361Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6289675Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6289682Z 2022-11-23T02:00:45.6289897Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6289979Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6289984Z 2022-11-23T02:00:45.6290058Z OK (skipped=1) 2022-11-23T02:00:45.6290063Z 2022-11-23T02:00:45.6290154Z Generating XML reports... 2022-11-23T02:00:45.6290513Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015035.xml 2022-11-23T02:00:45.6290770Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6291069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6291204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6291509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6291658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6291663Z 2022-11-23T02:00:45.6291743Z Running tests... 2022-11-23T02:00:45.6291960Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6292268Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6292274Z 2022-11-23T02:00:45.6292487Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6292566Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6292608Z 2022-11-23T02:00:45.6292793Z OK (skipped=1) 2022-11-23T02:00:45.6292797Z 2022-11-23T02:00:45.6292880Z Generating XML reports... 2022-11-23T02:00:45.6293227Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015038.xml 2022-11-23T02:00:45.6293472Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6293758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6293885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6294175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6294312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6294317Z 2022-11-23T02:00:45.6294389Z Running tests... 2022-11-23T02:00:45.6294591Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6294932Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6294973Z 2022-11-23T02:00:45.6295173Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6295244Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6295249Z 2022-11-23T02:00:45.6295320Z OK (skipped=1) 2022-11-23T02:00:45.6295325Z 2022-11-23T02:00:45.6295398Z Generating XML reports... 2022-11-23T02:00:45.6295744Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015041.xml 2022-11-23T02:00:45.6295985Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6296267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6296400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6296695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6296838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6296843Z 2022-11-23T02:00:45.6296919Z Running tests... 2022-11-23T02:00:45.6297122Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6297453Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6297459Z 2022-11-23T02:00:45.6297661Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6297739Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6297744Z 2022-11-23T02:00:45.6297818Z OK (skipped=1) 2022-11-23T02:00:45.6297822Z 2022-11-23T02:00:45.6297908Z Generating XML reports... 2022-11-23T02:00:45.6298251Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015044.xml 2022-11-23T02:00:45.6298493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6298774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6298899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6299189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6299324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6299329Z 2022-11-23T02:00:45.6299439Z Running tests... 2022-11-23T02:00:45.6299639Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6299977Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6299982Z 2022-11-23T02:00:45.6300181Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6300259Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6300264Z 2022-11-23T02:00:45.6300335Z OK (skipped=1) 2022-11-23T02:00:45.6300339Z 2022-11-23T02:00:45.6300412Z Generating XML reports... 2022-11-23T02:00:45.6300761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015047.xml 2022-11-23T02:00:45.6300999Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6301282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6301405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6301737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6301878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6301882Z 2022-11-23T02:00:45.6301955Z Running tests... 2022-11-23T02:00:45.6302159Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6302489Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6302495Z 2022-11-23T02:00:45.6302696Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6302775Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6302779Z 2022-11-23T02:00:45.6302849Z OK (skipped=1) 2022-11-23T02:00:45.6302853Z 2022-11-23T02:00:45.6302938Z Generating XML reports... 2022-11-23T02:00:45.6303287Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015050.xml 2022-11-23T02:00:45.6303525Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6303807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6303936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6304226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6304361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6304368Z 2022-11-23T02:00:45.6304438Z Running tests... 2022-11-23T02:00:45.6304639Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6304968Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6304974Z 2022-11-23T02:00:45.6305171Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6305246Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6305250Z 2022-11-23T02:00:45.6305310Z OK (skipped=1) 2022-11-23T02:00:45.6305327Z 2022-11-23T02:00:45.6305399Z Generating XML reports... 2022-11-23T02:00:45.6305747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015053.xml 2022-11-23T02:00:45.6305990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6306320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6306450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6306749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6306887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6306892Z 2022-11-23T02:00:45.6306965Z Running tests... 2022-11-23T02:00:45.6307164Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6307497Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6307502Z 2022-11-23T02:00:45.6307704Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6307781Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6307785Z 2022-11-23T02:00:45.6307856Z OK (skipped=1) 2022-11-23T02:00:45.6307860Z 2022-11-23T02:00:45.6307944Z Generating XML reports... 2022-11-23T02:00:45.6308327Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015056.xml 2022-11-23T02:00:45.6308571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6308856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6308983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6309272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6309410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6309417Z 2022-11-23T02:00:45.6309487Z Running tests... 2022-11-23T02:00:45.6309688Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6310020Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6310026Z 2022-11-23T02:00:45.6310224Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6310300Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6310305Z 2022-11-23T02:00:45.6310364Z OK (skipped=1) 2022-11-23T02:00:45.6310382Z 2022-11-23T02:00:45.6310454Z Generating XML reports... 2022-11-23T02:00:45.6310802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015059.xml 2022-11-23T02:00:45.6311046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6311333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6311460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6311757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6311894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6311899Z 2022-11-23T02:00:45.6311974Z Running tests... 2022-11-23T02:00:45.6312178Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6312509Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6312552Z 2022-11-23T02:00:45.6312752Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6312828Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6312833Z 2022-11-23T02:00:45.6312903Z OK (skipped=1) 2022-11-23T02:00:45.6312907Z 2022-11-23T02:00:45.6312994Z Generating XML reports... 2022-11-23T02:00:45.6313340Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015102.xml 2022-11-23T02:00:45.6313579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6313865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6313987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6314278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6314417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6314422Z 2022-11-23T02:00:45.6314496Z Running tests... 2022-11-23T02:00:45.6314701Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6315024Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6315030Z 2022-11-23T02:00:45.6315229Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6315293Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6315313Z 2022-11-23T02:00:45.6315372Z OK (skipped=1) 2022-11-23T02:00:45.6315377Z 2022-11-23T02:00:45.6315461Z Generating XML reports... 2022-11-23T02:00:45.6315813Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015105.xml 2022-11-23T02:00:45.6316054Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6316339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6316467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6316763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6316899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6316905Z 2022-11-23T02:00:45.6316979Z Running tests... 2022-11-23T02:00:45.6317178Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6317464Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:00:45.6317470Z 2022-11-23T02:00:45.6317668Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6317743Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6317748Z 2022-11-23T02:00:45.6317823Z OK (skipped=1) 2022-11-23T02:00:45.6317827Z 2022-11-23T02:00:45.6317914Z Generating XML reports... 2022-11-23T02:00:45.6318261Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015108.xml 2022-11-23T02:00:45.6318503Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6318790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6318918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6319207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6319344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6319388Z 2022-11-23T02:00:45.6319463Z Running tests... 2022-11-23T02:00:45.6319654Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6319991Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:00:45.6320007Z 2022-11-23T02:00:45.6320252Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6320503Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6320509Z 2022-11-23T02:00:45.6320758Z OK (skipped=1) 2022-11-23T02:00:45.6320763Z 2022-11-23T02:00:45.6320858Z Generating XML reports... 2022-11-23T02:00:45.6321241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015111.xml 2022-11-23T02:00:45.6321503Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6321997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6322133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6322450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6322654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6322659Z 2022-11-23T02:00:45.6322736Z Running tests... 2022-11-23T02:00:45.6322955Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6323207Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19404 2022-11-23T02:00:45.6323381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19405 2022-11-23T02:00:45.6323599Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6323910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6324047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6324370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6324523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6324725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6325031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6325170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6325472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6325628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6325820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6326161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6326499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6326683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6326862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6327071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse3ky0cj 2022-11-23T02:00:45.6327291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse3ky0cj/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6327542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8r383py 2022-11-23T02:00:45.6327758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8r383py/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6327835Z ok (7.575s) 2022-11-23T02:00:45.6327842Z 2022-11-23T02:00:45.6328071Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6328157Z Ran 1 test in 7.576s 2022-11-23T02:00:45.6328163Z 2022-11-23T02:00:45.6328225Z OK 2022-11-23T02:00:45.6328231Z 2022-11-23T02:00:45.6328321Z Generating XML reports... 2022-11-23T02:00:45.6328702Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015114.xml 2022-11-23T02:00:45.6328961Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6329270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6329414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6329734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6329921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6329927Z 2022-11-23T02:00:45.6329995Z Running tests... 2022-11-23T02:00:45.6330220Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6330488Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19593 2022-11-23T02:00:45.6330667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19594 2022-11-23T02:00:45.6330884Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6331195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6331442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6331754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6331898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6332089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6332388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6332522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6332923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6333058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6333244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6333554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6333862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6334033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6334203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6334396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposgkna_e 2022-11-23T02:00:45.6334602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposgkna_e/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6339909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51jy10fr 2022-11-23T02:00:45.6340367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51jy10fr/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6340602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6340840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6341326Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:00:45.6341464Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:00:45.6341768Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:00:45.6341880Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:00:45.6341949Z ok (6.481s) 2022-11-23T02:00:45.6341956Z 2022-11-23T02:00:45.6342161Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6342229Z Ran 1 test in 6.481s 2022-11-23T02:00:45.6342235Z 2022-11-23T02:00:45.6342293Z OK 2022-11-23T02:00:45.6342297Z 2022-11-23T02:00:45.6342381Z Generating XML reports... 2022-11-23T02:00:45.6342807Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015125.xml 2022-11-23T02:00:45.6343048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6343332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6343457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6343748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6343885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6343893Z 2022-11-23T02:00:45.6343968Z Running tests... 2022-11-23T02:00:45.6344171Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6344376Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.005s) 2022-11-23T02:00:45.6344381Z 2022-11-23T02:00:45.6344584Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6344658Z Ran 1 test in 0.005s 2022-11-23T02:00:45.6344663Z 2022-11-23T02:00:45.6344730Z OK (skipped=1) 2022-11-23T02:00:45.6344735Z 2022-11-23T02:00:45.6344816Z Generating XML reports... 2022-11-23T02:00:45.6345164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015134.xml 2022-11-23T02:00:45.6345403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6345682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6345807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6346092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6346227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6346232Z 2022-11-23T02:00:45.6346294Z Running tests... 2022-11-23T02:00:45.6346497Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6346735Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19847 2022-11-23T02:00:45.6346895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19848 2022-11-23T02:00:45.6347092Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6347443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6347569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6347859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6348001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6348185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6348472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6348599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6348892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6349031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6349211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6349522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6349864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6350032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6350196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6350382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zlqiojy 2022-11-23T02:00:45.6350578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zlqiojy/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6350763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzndbbfj 2022-11-23T02:00:45.6350964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzndbbfj/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6351124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6351294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6351364Z ok (6.684s) 2022-11-23T02:00:45.6351369Z 2022-11-23T02:00:45.6351579Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6351654Z Ran 1 test in 6.684s 2022-11-23T02:00:45.6351659Z 2022-11-23T02:00:45.6351718Z OK 2022-11-23T02:00:45.6351723Z 2022-11-23T02:00:45.6351806Z Generating XML reports... 2022-11-23T02:00:45.6352156Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015138.xml 2022-11-23T02:00:45.6352395Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6352683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6352811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6353105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6353242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6353247Z 2022-11-23T02:00:45.6353320Z Running tests... 2022-11-23T02:00:45.6353525Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6353886Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:00:45.6353892Z 2022-11-23T02:00:45.6354091Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6354202Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6354207Z 2022-11-23T02:00:45.6354280Z OK (skipped=1) 2022-11-23T02:00:45.6354285Z 2022-11-23T02:00:45.6354367Z Generating XML reports... 2022-11-23T02:00:45.6354717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015147.xml 2022-11-23T02:00:45.6354956Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6355226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6355353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6355645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6355781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6355789Z 2022-11-23T02:00:45.6355861Z Running tests... 2022-11-23T02:00:45.6356061Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6356412Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:00:45.6356417Z 2022-11-23T02:00:45.6356650Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6356729Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6356734Z 2022-11-23T02:00:45.6356814Z OK (skipped=1) 2022-11-23T02:00:45.6356819Z 2022-11-23T02:00:45.6356906Z Generating XML reports... 2022-11-23T02:00:45.6357251Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015150.xml 2022-11-23T02:00:45.6357490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6357772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6357902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6358191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6358332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6358337Z 2022-11-23T02:00:45.6358412Z Running tests... 2022-11-23T02:00:45.6358610Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6358993Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:00:45.6358998Z 2022-11-23T02:00:45.6359195Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6359270Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6359275Z 2022-11-23T02:00:45.6359347Z OK (skipped=1) 2022-11-23T02:00:45.6359352Z 2022-11-23T02:00:45.6359434Z Generating XML reports... 2022-11-23T02:00:45.6359770Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015153.xml 2022-11-23T02:00:45.6360009Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6360555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6360689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6361002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6361148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6361154Z 2022-11-23T02:00:45.6361233Z Running tests... 2022-11-23T02:00:45.6361447Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6361910Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:00:45.6361916Z 2022-11-23T02:00:45.6362127Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6362209Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6362214Z 2022-11-23T02:00:45.6362291Z OK (skipped=1) 2022-11-23T02:00:45.6362295Z 2022-11-23T02:00:45.6362389Z Generating XML reports... 2022-11-23T02:00:45.6362754Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015156.xml 2022-11-23T02:00:45.6363004Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6363306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6363441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6363747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6363890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6363895Z 2022-11-23T02:00:45.6364011Z Running tests... 2022-11-23T02:00:45.6364225Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6364566Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:00:45.6364572Z 2022-11-23T02:00:45.6364779Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6364848Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6364860Z 2022-11-23T02:00:45.6364923Z OK (skipped=1) 2022-11-23T02:00:45.6364928Z 2022-11-23T02:00:45.6365014Z Generating XML reports... 2022-11-23T02:00:45.6365382Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015159.xml 2022-11-23T02:00:45.6365634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6365935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6366070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6366379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6366531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6366536Z 2022-11-23T02:00:45.6366617Z Running tests... 2022-11-23T02:00:45.6366831Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6367087Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20333 2022-11-23T02:00:45.6367257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20334 2022-11-23T02:00:45.6367470Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6367767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6367897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6368201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6368344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6368528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6368825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6368996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6369305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6369452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6369631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6369957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6370277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6370451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6370627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6370826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5vkf203 2022-11-23T02:00:45.6371034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5vkf203/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6371268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1v8a2u4g 2022-11-23T02:00:45.6371477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1v8a2u4g/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6372112Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:00:45.6372832Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:00:45.6373011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6373182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6373253Z ok (6.476s) 2022-11-23T02:00:45.6373259Z 2022-11-23T02:00:45.6373470Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6373552Z Ran 1 test in 6.477s 2022-11-23T02:00:45.6373557Z 2022-11-23T02:00:45.6373621Z OK 2022-11-23T02:00:45.6373626Z 2022-11-23T02:00:45.6373708Z Generating XML reports... 2022-11-23T02:00:45.6374059Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015203.xml 2022-11-23T02:00:45.6374299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6374581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6374707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6374994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6375131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6375172Z 2022-11-23T02:00:45.6375243Z Running tests... 2022-11-23T02:00:45.6375447Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6376166Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.617s) 2022-11-23T02:00:45.6376173Z 2022-11-23T02:00:45.6376370Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6376448Z Ran 1 test in 0.617s 2022-11-23T02:00:45.6376453Z 2022-11-23T02:00:45.6376530Z OK (skipped=1) 2022-11-23T02:00:45.6376534Z 2022-11-23T02:00:45.6376619Z Generating XML reports... 2022-11-23T02:00:45.6376967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015212.xml 2022-11-23T02:00:45.6377212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6377494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6377644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6377937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6378075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6378080Z 2022-11-23T02:00:45.6378152Z Running tests... 2022-11-23T02:00:45.6378355Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6378706Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:00:45.6378714Z 2022-11-23T02:00:45.6378916Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6378994Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6378999Z 2022-11-23T02:00:45.6379072Z OK (skipped=1) 2022-11-23T02:00:45.6379076Z 2022-11-23T02:00:45.6379163Z Generating XML reports... 2022-11-23T02:00:45.6379513Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015216.xml 2022-11-23T02:00:45.6379751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6380029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6380153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6380442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6380578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6380584Z 2022-11-23T02:00:45.6380657Z Running tests... 2022-11-23T02:00:45.6380855Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6381204Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:00:45.6381209Z 2022-11-23T02:00:45.6381411Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6381489Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6381493Z 2022-11-23T02:00:45.6381561Z OK (skipped=1) 2022-11-23T02:00:45.6381566Z 2022-11-23T02:00:45.6381649Z Generating XML reports... 2022-11-23T02:00:45.6381997Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015219.xml 2022-11-23T02:00:45.6382222Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6382541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6382665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6382957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6383092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6383096Z 2022-11-23T02:00:45.6383168Z Running tests... 2022-11-23T02:00:45.6383368Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6383609Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20703 2022-11-23T02:00:45.6383771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20704 2022-11-23T02:00:45.6383970Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6384252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6384379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6384703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6384837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6385015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6385298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6385424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6385717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6385858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6386040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6386351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6386658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6386822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6386974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6387162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5wx66kj 2022-11-23T02:00:45.6387361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5wx66kj/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6387545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cdhjttl 2022-11-23T02:00:45.6387741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cdhjttl/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6387809Z ok (4.557s) 2022-11-23T02:00:45.6387817Z 2022-11-23T02:00:45.6388023Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6388101Z Ran 1 test in 4.558s 2022-11-23T02:00:45.6388106Z 2022-11-23T02:00:45.6388164Z OK 2022-11-23T02:00:45.6388168Z 2022-11-23T02:00:45.6388255Z Generating XML reports... 2022-11-23T02:00:45.6388599Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015222.xml 2022-11-23T02:00:45.6388837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6389118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6389283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6389576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6389713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6389718Z 2022-11-23T02:00:45.6389793Z Running tests... 2022-11-23T02:00:45.6389995Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6390717Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.631s) 2022-11-23T02:00:45.6390725Z 2022-11-23T02:00:45.6390929Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6391006Z Ran 1 test in 0.631s 2022-11-23T02:00:45.6391010Z 2022-11-23T02:00:45.6391086Z OK (skipped=1) 2022-11-23T02:00:45.6391090Z 2022-11-23T02:00:45.6391176Z Generating XML reports... 2022-11-23T02:00:45.6391563Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015229.xml 2022-11-23T02:00:45.6391805Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6392075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6392198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6392487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6392630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6392635Z 2022-11-23T02:00:45.6392709Z Running tests... 2022-11-23T02:00:45.6392909Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6393262Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:00:45.6393268Z 2022-11-23T02:00:45.6393461Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6393534Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6393539Z 2022-11-23T02:00:45.6393610Z OK (skipped=1) 2022-11-23T02:00:45.6393615Z 2022-11-23T02:00:45.6393699Z Generating XML reports... 2022-11-23T02:00:45.6394046Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015233.xml 2022-11-23T02:00:45.6394287Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6394575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6394705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6394996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6395135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6395140Z 2022-11-23T02:00:45.6395213Z Running tests... 2022-11-23T02:00:45.6395416Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6396133Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.517s) 2022-11-23T02:00:45.6396175Z 2022-11-23T02:00:45.6396374Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6396449Z Ran 1 test in 0.517s 2022-11-23T02:00:45.6396454Z 2022-11-23T02:00:45.6396525Z OK (skipped=1) 2022-11-23T02:00:45.6396529Z 2022-11-23T02:00:45.6396614Z Generating XML reports... 2022-11-23T02:00:45.6396959Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015236.xml 2022-11-23T02:00:45.6397194Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6397467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6397589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6397877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6398017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6398022Z 2022-11-23T02:00:45.6398093Z Running tests... 2022-11-23T02:00:45.6398294Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6398585Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21063 2022-11-23T02:00:45.6398744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21064 2022-11-23T02:00:45.6398942Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6399226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6399349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6399639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6399777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6399957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6400302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6400604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6400914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6401062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6401249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6401572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6401898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6402077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6402254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6402440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0weavcng 2022-11-23T02:00:45.6402648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0weavcng/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6402842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vt9e6qg 2022-11-23T02:00:45.6403051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vt9e6qg/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6403415Z STAGE:2022-11-23 01:52:42 21063:21063 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6403677Z STAGE:2022-11-23 01:52:42 21064:21064 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6404201Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.6404280Z warnings.warn( 2022-11-23T02:00:45.6404457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6404960Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:00:45.6405041Z warnings.warn( 2022-11-23T02:00:45.6405225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:00:45.6405504Z STAGE:2022-11-23 01:52:43 21063:21063 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6405777Z STAGE:2022-11-23 01:52:43 21064:21064 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6408328Z STAGE:2022-11-23 01:52:43 21063:21063 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6408624Z STAGE:2022-11-23 01:52:43 21064:21064 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6408808Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:00:45.6408992Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:00:45.6409258Z STAGE:2022-11-23 01:52:43 21063:21063 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6409531Z STAGE:2022-11-23 01:52:43 21063:21063 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6409811Z STAGE:2022-11-23 01:52:43 21063:21063 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6409883Z ok (4.946s) 2022-11-23T02:00:45.6409889Z 2022-11-23T02:00:45.6410108Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6410177Z Ran 1 test in 4.946s 2022-11-23T02:00:45.6410192Z 2022-11-23T02:00:45.6410245Z OK 2022-11-23T02:00:45.6410249Z 2022-11-23T02:00:45.6410333Z Generating XML reports... 2022-11-23T02:00:45.6410697Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015240.xml 2022-11-23T02:00:45.6410953Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6411250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6411385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6411691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6411835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6411843Z 2022-11-23T02:00:45.6411923Z Running tests... 2022-11-23T02:00:45.6412136Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6412390Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21263 2022-11-23T02:00:45.6412561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21264 2022-11-23T02:00:45.6412773Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6413073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6413263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6413567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6413709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6413895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6414193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6414324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6414628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6414771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6414950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6415274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6415630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6415805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6415980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6416178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_zl9jxy 2022-11-23T02:00:45.6416388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_zl9jxy/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6416584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpos1g2y39 2022-11-23T02:00:45.6416792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpos1g2y39/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6416866Z ok (4.442s) 2022-11-23T02:00:45.6416871Z 2022-11-23T02:00:45.6417091Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6417179Z Ran 1 test in 4.442s 2022-11-23T02:00:45.6417184Z 2022-11-23T02:00:45.6417247Z OK 2022-11-23T02:00:45.6417252Z 2022-11-23T02:00:45.6417342Z Generating XML reports... 2022-11-23T02:00:45.6417708Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015248.xml 2022-11-23T02:00:45.6417965Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6418264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6418397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6418706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6418850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6418855Z 2022-11-23T02:00:45.6418937Z Running tests... 2022-11-23T02:00:45.6419153Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6419400Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21453 2022-11-23T02:00:45.6419566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21454 2022-11-23T02:00:45.6419774Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6420075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6420242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6420655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6420793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6420975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6421255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6421378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6421666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6421800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6422149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6422480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6422849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6423026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6423200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6423398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw3kyhjd 2022-11-23T02:00:45.6423607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw3kyhjd/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6423803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nc_qu9k 2022-11-23T02:00:45.6424014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nc_qu9k/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6424084Z ok (4.341s) 2022-11-23T02:00:45.6424089Z 2022-11-23T02:00:45.6424304Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6424372Z Ran 1 test in 4.341s 2022-11-23T02:00:45.6424376Z 2022-11-23T02:00:45.6424438Z OK 2022-11-23T02:00:45.6424444Z 2022-11-23T02:00:45.6424527Z Generating XML reports... 2022-11-23T02:00:45.6424891Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015255.xml 2022-11-23T02:00:45.6425137Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6425609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6425738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6426046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6426194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6426200Z 2022-11-23T02:00:45.6426276Z Running tests... 2022-11-23T02:00:45.6426490Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6427243Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.512s) 2022-11-23T02:00:45.6427249Z 2022-11-23T02:00:45.6427458Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6427539Z Ran 1 test in 0.513s 2022-11-23T02:00:45.6427583Z 2022-11-23T02:00:45.6427657Z OK (skipped=1) 2022-11-23T02:00:45.6427662Z 2022-11-23T02:00:45.6427750Z Generating XML reports... 2022-11-23T02:00:45.6428117Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015302.xml 2022-11-23T02:00:45.6428372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6428668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6428801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6429105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6429246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6429252Z 2022-11-23T02:00:45.6429329Z Running tests... 2022-11-23T02:00:45.6429540Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6430311Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.514s) 2022-11-23T02:00:45.6430319Z 2022-11-23T02:00:45.6430532Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6430614Z Ran 1 test in 0.514s 2022-11-23T02:00:45.6430619Z 2022-11-23T02:00:45.6430690Z OK (skipped=1) 2022-11-23T02:00:45.6430695Z 2022-11-23T02:00:45.6430772Z Generating XML reports... 2022-11-23T02:00:45.6431140Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015306.xml 2022-11-23T02:00:45.6431393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6431694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6431824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6432131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6432278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6432282Z 2022-11-23T02:00:45.6432358Z Running tests... 2022-11-23T02:00:45.6432572Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6433337Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.511s) 2022-11-23T02:00:45.6433345Z 2022-11-23T02:00:45.6433558Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6433640Z Ran 1 test in 0.511s 2022-11-23T02:00:45.6433645Z 2022-11-23T02:00:45.6433719Z OK (skipped=1) 2022-11-23T02:00:45.6433726Z 2022-11-23T02:00:45.6433817Z Generating XML reports... 2022-11-23T02:00:45.6434183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015309.xml 2022-11-23T02:00:45.6434436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6434735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6434868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6435172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6435356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6435361Z 2022-11-23T02:00:45.6435443Z Running tests... 2022-11-23T02:00:45.6435654Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6436167Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:00:45.6436173Z 2022-11-23T02:00:45.6436370Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6436443Z Ran 1 test in 0.003s 2022-11-23T02:00:45.6436449Z 2022-11-23T02:00:45.6436519Z OK (skipped=1) 2022-11-23T02:00:45.6436524Z 2022-11-23T02:00:45.6436595Z Generating XML reports... 2022-11-23T02:00:45.6436942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015313.xml 2022-11-23T02:00:45.6437183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6437465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6437625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6437917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6438051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6438056Z 2022-11-23T02:00:45.6438128Z Running tests... 2022-11-23T02:00:45.6438331Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6438569Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21871 2022-11-23T02:00:45.6438729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21872 2022-11-23T02:00:45.6438927Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6439209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6439335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6439628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6439760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6439937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6440295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6440590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6440897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6441039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6441229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6441552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6441860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6442031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6442206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6442510Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:00:45.6442765Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:00:45.6442961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijpiqxh6 2022-11-23T02:00:45.6443175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijpiqxh6/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6443478Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:00:45.6443681Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:00:45.6443874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfvflbdt 2022-11-23T02:00:45.6444082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfvflbdt/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6444155Z ok (4.436s) 2022-11-23T02:00:45.6444161Z 2022-11-23T02:00:45.6444376Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6444454Z Ran 1 test in 4.437s 2022-11-23T02:00:45.6444460Z 2022-11-23T02:00:45.6444523Z OK 2022-11-23T02:00:45.6444528Z 2022-11-23T02:00:45.6444613Z Generating XML reports... 2022-11-23T02:00:45.6445022Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015316.xml 2022-11-23T02:00:45.6445277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6445575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6445706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6446010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6446153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6446161Z 2022-11-23T02:00:45.6446224Z Running tests... 2022-11-23T02:00:45.6446437Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6446682Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22057 2022-11-23T02:00:45.6446851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22058 2022-11-23T02:00:45.6447061Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6447355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6447486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6447790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6447939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6448130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6448428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6448564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6448870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6449012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6449198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6449522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6449886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6450060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6450244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.6450413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6450592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.6450910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6451224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6451284Z ok (3.841s) 2022-11-23T02:00:45.6451300Z 2022-11-23T02:00:45.6451506Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6451584Z Ran 1 test in 3.841s 2022-11-23T02:00:45.6451590Z 2022-11-23T02:00:45.6451652Z OK 2022-11-23T02:00:45.6451657Z 2022-11-23T02:00:45.6451745Z Generating XML reports... 2022-11-23T02:00:45.6452151Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015323.xml 2022-11-23T02:00:45.6452513Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6452799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6452927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6453216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6453354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6453362Z 2022-11-23T02:00:45.6453437Z Running tests... 2022-11-23T02:00:45.6453640Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6453870Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22238 2022-11-23T02:00:45.6454027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22239 2022-11-23T02:00:45.6454226Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6454508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6454632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6454923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6455066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6455243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6455525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6455639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6455926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6456058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6456235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6456545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6456846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6457067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6457240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.6457400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6457569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.6457872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6458172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6458243Z ok (3.841s) 2022-11-23T02:00:45.6458248Z 2022-11-23T02:00:45.6458450Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6458527Z Ran 1 test in 3.842s 2022-11-23T02:00:45.6458532Z 2022-11-23T02:00:45.6458591Z OK 2022-11-23T02:00:45.6458596Z 2022-11-23T02:00:45.6458680Z Generating XML reports... 2022-11-23T02:00:45.6459069Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015330.xml 2022-11-23T02:00:45.6459312Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6459600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6459725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6460014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6460151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6460158Z 2022-11-23T02:00:45.6460220Z Running tests... 2022-11-23T02:00:45.6460429Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6461148Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.530s) 2022-11-23T02:00:45.6461166Z 2022-11-23T02:00:45.6461355Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6461426Z Ran 1 test in 0.530s 2022-11-23T02:00:45.6461431Z 2022-11-23T02:00:45.6461499Z OK (skipped=1) 2022-11-23T02:00:45.6461503Z 2022-11-23T02:00:45.6461585Z Generating XML reports... 2022-11-23T02:00:45.6461929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015337.xml 2022-11-23T02:00:45.6462172Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6462455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6462584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6462875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6463010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6463015Z 2022-11-23T02:00:45.6463090Z Running tests... 2022-11-23T02:00:45.6463289Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6463634Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:00:45.6463682Z 2022-11-23T02:00:45.6463883Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6463955Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6463960Z 2022-11-23T02:00:45.6464028Z OK (skipped=1) 2022-11-23T02:00:45.6464032Z 2022-11-23T02:00:45.6464114Z Generating XML reports... 2022-11-23T02:00:45.6464459Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015341.xml 2022-11-23T02:00:45.6464699Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6464974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6465097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6465384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6465521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6465526Z 2022-11-23T02:00:45.6465588Z Running tests... 2022-11-23T02:00:45.6465793Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6466076Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22535 2022-11-23T02:00:45.6466236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22536 2022-11-23T02:00:45.6466432Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6466713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6466834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6467124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6467261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6467441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6467719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6467843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6468133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6468268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6468444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6468752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6469061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6469228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6469397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6469465Z ok (3.849s) 2022-11-23T02:00:45.6469469Z 2022-11-23T02:00:45.6469673Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6469813Z Ran 1 test in 3.850s 2022-11-23T02:00:45.6469818Z 2022-11-23T02:00:45.6469867Z OK 2022-11-23T02:00:45.6469882Z 2022-11-23T02:00:45.6469954Z Generating XML reports... 2022-11-23T02:00:45.6470300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015344.xml 2022-11-23T02:00:45.6470539Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6470869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6470996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6471289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6471423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6471428Z 2022-11-23T02:00:45.6471498Z Running tests... 2022-11-23T02:00:45.6471698Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6471885Z test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.6471890Z 2022-11-23T02:00:45.6472089Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6472166Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6472173Z 2022-11-23T02:00:45.6472248Z OK (skipped=1) 2022-11-23T02:00:45.6472252Z 2022-11-23T02:00:45.6472336Z Generating XML reports... 2022-11-23T02:00:45.6472682Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015350.xml 2022-11-23T02:00:45.6472953Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6473238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6473362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6473647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6473781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6473786Z 2022-11-23T02:00:45.6473854Z Running tests... 2022-11-23T02:00:45.6474047Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6474239Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6474254Z 2022-11-23T02:00:45.6474443Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6474518Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6474523Z 2022-11-23T02:00:45.6474595Z OK (skipped=1) 2022-11-23T02:00:45.6474599Z 2022-11-23T02:00:45.6474682Z Generating XML reports... 2022-11-23T02:00:45.6475024Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015353.xml 2022-11-23T02:00:45.6475264Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6475543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6475672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6475965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6476103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6476108Z 2022-11-23T02:00:45.6476180Z Running tests... 2022-11-23T02:00:45.6476382Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6476615Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22830 2022-11-23T02:00:45.6476771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22831 2022-11-23T02:00:45.6476971Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6477251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6477412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6477706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6477844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6478027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6478307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6478421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6478708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6478843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6479017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6479324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6479662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6479825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6479989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6480298Z STAGE:2022-11-23 01:53:59 22831:22831 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6480735Z STAGE:2022-11-23 01:53:59 22830:22830 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6481007Z STAGE:2022-11-23 01:53:59 22830:22830 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6481294Z STAGE:2022-11-23 01:53:59 22830:22830 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6481473Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6481747Z STAGE:2022-11-23 01:53:59 22831:22831 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6482032Z STAGE:2022-11-23 01:53:59 22831:22831 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6482211Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6482476Z STAGE:2022-11-23 01:53:59 22830:22830 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6482736Z STAGE:2022-11-23 01:53:59 22831:22831 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6483004Z STAGE:2022-11-23 01:53:59 22831:22831 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6483285Z STAGE:2022-11-23 01:53:59 22831:22831 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6483552Z STAGE:2022-11-23 01:53:59 22830:22830 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6483835Z STAGE:2022-11-23 01:53:59 22830:22830 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6483908Z ok (4.942s) 2022-11-23T02:00:45.6483913Z 2022-11-23T02:00:45.6484124Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6484194Z Ran 1 test in 4.942s 2022-11-23T02:00:45.6484199Z 2022-11-23T02:00:45.6484265Z OK 2022-11-23T02:00:45.6484270Z 2022-11-23T02:00:45.6484360Z Generating XML reports... 2022-11-23T02:00:45.6484726Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015356.xml 2022-11-23T02:00:45.6484976Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6485326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6485458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6485767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6485915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6485920Z 2022-11-23T02:00:45.6485997Z Running tests... 2022-11-23T02:00:45.6486211Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6486416Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6486422Z 2022-11-23T02:00:45.6486632Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6486714Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6486722Z 2022-11-23T02:00:45.6486799Z OK (skipped=1) 2022-11-23T02:00:45.6486804Z 2022-11-23T02:00:45.6486895Z Generating XML reports... 2022-11-23T02:00:45.6487263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015404.xml 2022-11-23T02:00:45.6487553Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6487853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6487982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6488284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6488430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6488435Z 2022-11-23T02:00:45.6488500Z Running tests... 2022-11-23T02:00:45.6488716Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6488913Z test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6488918Z 2022-11-23T02:00:45.6489127Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6489205Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6489211Z 2022-11-23T02:00:45.6489284Z OK (skipped=1) 2022-11-23T02:00:45.6489289Z 2022-11-23T02:00:45.6489375Z Generating XML reports... 2022-11-23T02:00:45.6489743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015407.xml 2022-11-23T02:00:45.6489988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6490283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6490413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6490718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6490863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6490868Z 2022-11-23T02:00:45.6490941Z Running tests... 2022-11-23T02:00:45.6491153Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6491396Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23138 2022-11-23T02:00:45.6491561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23139 2022-11-23T02:00:45.6491767Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6492064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6492343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6492632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6492771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6492940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6493223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6493346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6493634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6493771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6493951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6494265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6494606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6494773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6494937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6495006Z ok (4.556s) 2022-11-23T02:00:45.6495011Z 2022-11-23T02:00:45.6495213Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6495285Z Ran 1 test in 4.556s 2022-11-23T02:00:45.6495290Z 2022-11-23T02:00:45.6495349Z OK 2022-11-23T02:00:45.6495353Z 2022-11-23T02:00:45.6495436Z Generating XML reports... 2022-11-23T02:00:45.6495783Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015410.xml 2022-11-23T02:00:45.6496022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6496303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6496430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6496721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6496858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6496862Z 2022-11-23T02:00:45.6496933Z Running tests... 2022-11-23T02:00:45.6497123Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6497838Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.531s) 2022-11-23T02:00:45.6497845Z 2022-11-23T02:00:45.6498048Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6498120Z Ran 1 test in 0.531s 2022-11-23T02:00:45.6498124Z 2022-11-23T02:00:45.6498183Z OK (skipped=1) 2022-11-23T02:00:45.6498199Z 2022-11-23T02:00:45.6498272Z Generating XML reports... 2022-11-23T02:00:45.6498616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015418.xml 2022-11-23T02:00:45.6498859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6499144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6499308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6499601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6499736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6499744Z 2022-11-23T02:00:45.6499816Z Running tests... 2022-11-23T02:00:45.6500014Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6500240Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23382 2022-11-23T02:00:45.6500399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23383 2022-11-23T02:00:45.6500597Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6500877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6501006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6501300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6501493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6501675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6501961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6502083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6502368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6502499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6502666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6502974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6503280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6503442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6503616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.6503775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6503946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.6504247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6504546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6504613Z ok (3.864s) 2022-11-23T02:00:45.6504619Z 2022-11-23T02:00:45.6504819Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6504892Z Ran 1 test in 3.864s 2022-11-23T02:00:45.6504897Z 2022-11-23T02:00:45.6504959Z OK 2022-11-23T02:00:45.6504963Z 2022-11-23T02:00:45.6505044Z Generating XML reports... 2022-11-23T02:00:45.6505392Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015422.xml 2022-11-23T02:00:45.6505630Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6505916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6506077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6506371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6506509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6506517Z 2022-11-23T02:00:45.6506588Z Running tests... 2022-11-23T02:00:45.6506789Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6507017Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23563 2022-11-23T02:00:45.6507165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23564 2022-11-23T02:00:45.6507361Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6507645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6507773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6508060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6508251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6508427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6508704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6508827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6509111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6509246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6509427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6509735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6510039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6510206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6510375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6510445Z ok (4.250s) 2022-11-23T02:00:45.6510450Z 2022-11-23T02:00:45.6510656Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6510731Z Ran 1 test in 4.251s 2022-11-23T02:00:45.6510736Z 2022-11-23T02:00:45.6510798Z OK 2022-11-23T02:00:45.6510803Z 2022-11-23T02:00:45.6510890Z Generating XML reports... 2022-11-23T02:00:45.6511237Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015429.xml 2022-11-23T02:00:45.6511466Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6511753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6511879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6512168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6512304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6512310Z 2022-11-23T02:00:45.6512378Z Running tests... 2022-11-23T02:00:45.6512581Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6512806Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23749 2022-11-23T02:00:45.6513007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23750 2022-11-23T02:00:45.6513208Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6513492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6513612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6513897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6514035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6514214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6514496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6514624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6514913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6515079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6515257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6515568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6515875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6516043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6516200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6516266Z ok (3.950s) 2022-11-23T02:00:45.6516272Z 2022-11-23T02:00:45.6516474Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6516553Z Ran 1 test in 3.950s 2022-11-23T02:00:45.6516558Z 2022-11-23T02:00:45.6516622Z OK 2022-11-23T02:00:45.6516627Z 2022-11-23T02:00:45.6516710Z Generating XML reports... 2022-11-23T02:00:45.6517055Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015436.xml 2022-11-23T02:00:45.6517294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6517571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6517693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6517977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6518111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6518116Z 2022-11-23T02:00:45.6518186Z Running tests... 2022-11-23T02:00:45.6518390Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6518630Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23928 2022-11-23T02:00:45.6518788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23929 2022-11-23T02:00:45.6518984Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6519267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6519392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6519723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6519861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6520041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6520544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6520677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6520986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6521132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6521319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6521642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6521962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6522184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6522524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.6522696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6522873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.6523195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6523510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6523580Z ok (3.951s) 2022-11-23T02:00:45.6523585Z 2022-11-23T02:00:45.6523800Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6523876Z Ran 1 test in 3.952s 2022-11-23T02:00:45.6523882Z 2022-11-23T02:00:45.6523945Z OK 2022-11-23T02:00:45.6523951Z 2022-11-23T02:00:45.6524037Z Generating XML reports... 2022-11-23T02:00:45.6524409Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015443.xml 2022-11-23T02:00:45.6524664Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6524964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6525098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6525402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6525537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6525554Z 2022-11-23T02:00:45.6525618Z Running tests... 2022-11-23T02:00:45.6525834Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6526089Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24109 2022-11-23T02:00:45.6526257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24110 2022-11-23T02:00:45.6526463Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6526759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6526892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6527241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6527383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6527570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6527866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6527995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6528299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6528441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6528627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6528953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6529268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6529476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6529656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:00:45.6529827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6530009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:00:45.6530331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6530636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:00:45.6530711Z ok (3.852s) 2022-11-23T02:00:45.6530716Z 2022-11-23T02:00:45.6530928Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6531009Z Ran 1 test in 3.852s 2022-11-23T02:00:45.6531014Z 2022-11-23T02:00:45.6531079Z OK 2022-11-23T02:00:45.6531087Z 2022-11-23T02:00:45.6531176Z Generating XML reports... 2022-11-23T02:00:45.6531544Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015450.xml 2022-11-23T02:00:45.6531793Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6532202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6532324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6532617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6532753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6532758Z 2022-11-23T02:00:45.6532828Z Running tests... 2022-11-23T02:00:45.6533030Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6533364Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:00:45.6533370Z 2022-11-23T02:00:45.6533566Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6533630Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6533634Z 2022-11-23T02:00:45.6533707Z OK (skipped=1) 2022-11-23T02:00:45.6533711Z 2022-11-23T02:00:45.6533794Z Generating XML reports... 2022-11-23T02:00:45.6534139Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015457.xml 2022-11-23T02:00:45.6534422Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6534704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6534825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6535116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6535252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6535256Z 2022-11-23T02:00:45.6535328Z Running tests... 2022-11-23T02:00:45.6535530Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6535702Z test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s) 2022-11-23T02:00:45.6535707Z 2022-11-23T02:00:45.6535901Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6535975Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6535980Z 2022-11-23T02:00:45.6536049Z OK (skipped=1) 2022-11-23T02:00:45.6536053Z 2022-11-23T02:00:45.6536135Z Generating XML reports... 2022-11-23T02:00:45.6536521Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015500.xml 2022-11-23T02:00:45.6536761Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6537042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6537166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6537454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6537588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6537595Z 2022-11-23T02:00:45.6537656Z Running tests... 2022-11-23T02:00:45.6537854Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6538025Z test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:00:45.6538030Z 2022-11-23T02:00:45.6538227Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6538302Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6538307Z 2022-11-23T02:00:45.6538378Z OK (skipped=1) 2022-11-23T02:00:45.6538383Z 2022-11-23T02:00:45.6538470Z Generating XML reports... 2022-11-23T02:00:45.6538814Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015503.xml 2022-11-23T02:00:45.6539053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6539332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6539456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6539742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6539877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6539884Z 2022-11-23T02:00:45.6539956Z Running tests... 2022-11-23T02:00:45.6540158Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6540351Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:00:45.6540356Z 2022-11-23T02:00:45.6540549Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6540620Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6540626Z 2022-11-23T02:00:45.6540693Z OK (skipped=1) 2022-11-23T02:00:45.6540698Z 2022-11-23T02:00:45.6540781Z Generating XML reports... 2022-11-23T02:00:45.6541170Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015506.xml 2022-11-23T02:00:45.6541408Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6541683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6541814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6542108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6542247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6542251Z 2022-11-23T02:00:45.6542327Z Running tests... 2022-11-23T02:00:45.6542526Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6542718Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:00:45.6542725Z 2022-11-23T02:00:45.6542923Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6543001Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6543006Z 2022-11-23T02:00:45.6543080Z OK (skipped=1) 2022-11-23T02:00:45.6543085Z 2022-11-23T02:00:45.6543203Z Generating XML reports... 2022-11-23T02:00:45.6543553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015509.xml 2022-11-23T02:00:45.6543785Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6544066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6544189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6544476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6544609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6544614Z 2022-11-23T02:00:45.6544685Z Running tests... 2022-11-23T02:00:45.6544886Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6545237Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:00:45.6545242Z 2022-11-23T02:00:45.6545444Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6545519Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6545524Z 2022-11-23T02:00:45.6545595Z OK (skipped=1) 2022-11-23T02:00:45.6545599Z 2022-11-23T02:00:45.6545671Z Generating XML reports... 2022-11-23T02:00:45.6546020Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015512.xml 2022-11-23T02:00:45.6546260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6546547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6546674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6546967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6547105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6547111Z 2022-11-23T02:00:45.6547183Z Running tests... 2022-11-23T02:00:45.6547386Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6547760Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:00:45.6547766Z 2022-11-23T02:00:45.6547965Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6548076Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6548081Z 2022-11-23T02:00:45.6548154Z OK (skipped=1) 2022-11-23T02:00:45.6548158Z 2022-11-23T02:00:45.6548242Z Generating XML reports... 2022-11-23T02:00:45.6548592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015515.xml 2022-11-23T02:00:45.6548828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6549112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6549234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6549517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6549653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6549660Z 2022-11-23T02:00:45.6549731Z Running tests... 2022-11-23T02:00:45.6549931Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6550253Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.6550259Z 2022-11-23T02:00:45.6550500Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6550566Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6550583Z 2022-11-23T02:00:45.6550644Z OK (skipped=1) 2022-11-23T02:00:45.6550648Z 2022-11-23T02:00:45.6550731Z Generating XML reports... 2022-11-23T02:00:45.6551084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015518.xml 2022-11-23T02:00:45.6551324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6551608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6551736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6552026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6552166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6552171Z 2022-11-23T02:00:45.6552246Z Running tests... 2022-11-23T02:00:45.6552445Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6552749Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.6552754Z 2022-11-23T02:00:45.6552950Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6553022Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6553026Z 2022-11-23T02:00:45.6553095Z OK (skipped=1) 2022-11-23T02:00:45.6553101Z 2022-11-23T02:00:45.6553187Z Generating XML reports... 2022-11-23T02:00:45.6553534Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015521.xml 2022-11-23T02:00:45.6553774Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6554059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6554182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6554468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6554603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6554607Z 2022-11-23T02:00:45.6554679Z Running tests... 2022-11-23T02:00:45.6554870Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6555240Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.6555244Z 2022-11-23T02:00:45.6555443Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6555520Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6555525Z 2022-11-23T02:00:45.6555595Z OK (skipped=1) 2022-11-23T02:00:45.6555599Z 2022-11-23T02:00:45.6555678Z Generating XML reports... 2022-11-23T02:00:45.6556018Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015524.xml 2022-11-23T02:00:45.6556253Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6556528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6556650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6556942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6557075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6557081Z 2022-11-23T02:00:45.6557155Z Running tests... 2022-11-23T02:00:45.6557387Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6557710Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.6557715Z 2022-11-23T02:00:45.6557914Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6557989Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6557994Z 2022-11-23T02:00:45.6558064Z OK (skipped=1) 2022-11-23T02:00:45.6558068Z 2022-11-23T02:00:45.6558153Z Generating XML reports... 2022-11-23T02:00:45.6558502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015527.xml 2022-11-23T02:00:45.6558745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6559026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6559152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6559431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6559566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6559571Z 2022-11-23T02:00:45.6559643Z Running tests... 2022-11-23T02:00:45.6559843Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6560216Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.6560224Z 2022-11-23T02:00:45.6560601Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6560682Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6560687Z 2022-11-23T02:00:45.6560760Z OK (skipped=1) 2022-11-23T02:00:45.6560765Z 2022-11-23T02:00:45.6560855Z Generating XML reports... 2022-11-23T02:00:45.6561224Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015530.xml 2022-11-23T02:00:45.6561477Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6561779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6561910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6562219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6562364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6562432Z 2022-11-23T02:00:45.6562507Z Running tests... 2022-11-23T02:00:45.6562720Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6562979Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24986 2022-11-23T02:00:45.6563145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24987 2022-11-23T02:00:45.6563352Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6563651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6563778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6564086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6564220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6564408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6564747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6564881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6565184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6565325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6565511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6565838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6566157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6566336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6566510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6566584Z ok (4.332s) 2022-11-23T02:00:45.6566589Z 2022-11-23T02:00:45.6566803Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6566884Z Ran 1 test in 4.333s 2022-11-23T02:00:45.6566889Z 2022-11-23T02:00:45.6566953Z OK 2022-11-23T02:00:45.6566957Z 2022-11-23T02:00:45.6567045Z Generating XML reports... 2022-11-23T02:00:45.6567413Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015533.xml 2022-11-23T02:00:45.6567664Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6567965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6568102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6568412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6568555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6568561Z 2022-11-23T02:00:45.6568625Z Running tests... 2022-11-23T02:00:45.6568836Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6569095Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25176 2022-11-23T02:00:45.6569266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25177 2022-11-23T02:00:45.6569475Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6569807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6569935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6570240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6570380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6570568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6570862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6570990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6571291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6571434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6571619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6571983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6572421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6572590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6572755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6572824Z ok (4.653s) 2022-11-23T02:00:45.6572829Z 2022-11-23T02:00:45.6573036Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6573114Z Ran 1 test in 4.653s 2022-11-23T02:00:45.6573119Z 2022-11-23T02:00:45.6573168Z OK 2022-11-23T02:00:45.6573185Z 2022-11-23T02:00:45.6573257Z Generating XML reports... 2022-11-23T02:00:45.6573607Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015541.xml 2022-11-23T02:00:45.6573847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6574129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6574249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6574538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6574672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6574677Z 2022-11-23T02:00:45.6574750Z Running tests... 2022-11-23T02:00:45.6574948Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6575188Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25366 2022-11-23T02:00:45.6575346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25367 2022-11-23T02:00:45.6575539Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6575816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6575939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6576224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6576358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6576570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6576856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6576983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6577273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6577408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6577584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6577881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6578183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6578354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6578518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6578584Z ok (4.263s) 2022-11-23T02:00:45.6578590Z 2022-11-23T02:00:45.6578831Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6578906Z Ran 1 test in 4.264s 2022-11-23T02:00:45.6578910Z 2022-11-23T02:00:45.6578969Z OK 2022-11-23T02:00:45.6578973Z 2022-11-23T02:00:45.6579056Z Generating XML reports... 2022-11-23T02:00:45.6579406Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015549.xml 2022-11-23T02:00:45.6579643Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6579922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6580047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6580333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6580467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6580472Z 2022-11-23T02:00:45.6580540Z Running tests... 2022-11-23T02:00:45.6580740Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6580981Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25556 2022-11-23T02:00:45.6581139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25557 2022-11-23T02:00:45.6581336Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6581620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6581747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6582025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6582163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6582346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6582629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6582756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6583046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6583220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6583398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6583710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6584019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6584183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6584346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6584411Z ok (4.753s) 2022-11-23T02:00:45.6584416Z 2022-11-23T02:00:45.6584620Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6584693Z Ran 1 test in 4.754s 2022-11-23T02:00:45.6584698Z 2022-11-23T02:00:45.6584760Z OK 2022-11-23T02:00:45.6584765Z 2022-11-23T02:00:45.6584847Z Generating XML reports... 2022-11-23T02:00:45.6585193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015556.xml 2022-11-23T02:00:45.6585429Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6585735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6585857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6586148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6586272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6586287Z 2022-11-23T02:00:45.6586348Z Running tests... 2022-11-23T02:00:45.6586550Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6587268Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.519s) 2022-11-23T02:00:45.6587286Z 2022-11-23T02:00:45.6587475Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6587553Z Ran 1 test in 0.519s 2022-11-23T02:00:45.6587558Z 2022-11-23T02:00:45.6587629Z OK (skipped=1) 2022-11-23T02:00:45.6587634Z 2022-11-23T02:00:45.6587719Z Generating XML reports... 2022-11-23T02:00:45.6588062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015604.xml 2022-11-23T02:00:45.6588299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6588581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6588704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6588995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6589131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6589136Z 2022-11-23T02:00:45.6589206Z Running tests... 2022-11-23T02:00:45.6589403Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6589586Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:00:45.6589592Z 2022-11-23T02:00:45.6589789Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6589860Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6589901Z 2022-11-23T02:00:45.6589971Z OK (skipped=1) 2022-11-23T02:00:45.6589975Z 2022-11-23T02:00:45.6590054Z Generating XML reports... 2022-11-23T02:00:45.6590403Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015607.xml 2022-11-23T02:00:45.6590639Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6590917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6591040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6591331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6591457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6591474Z 2022-11-23T02:00:45.6591534Z Running tests... 2022-11-23T02:00:45.6591737Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6591941Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:00:45.6591946Z 2022-11-23T02:00:45.6592142Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6592216Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6592256Z 2022-11-23T02:00:45.6592326Z OK (skipped=1) 2022-11-23T02:00:45.6592331Z 2022-11-23T02:00:45.6592412Z Generating XML reports... 2022-11-23T02:00:45.6592758Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015611.xml 2022-11-23T02:00:45.6592994Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6593275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6593400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6593693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6593829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6593834Z 2022-11-23T02:00:45.6593905Z Running tests... 2022-11-23T02:00:45.6594108Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6594340Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:00:45.6594345Z 2022-11-23T02:00:45.6594546Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6594622Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6594626Z 2022-11-23T02:00:45.6594694Z OK (skipped=1) 2022-11-23T02:00:45.6594699Z 2022-11-23T02:00:45.6594783Z Generating XML reports... 2022-11-23T02:00:45.6595132Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015614.xml 2022-11-23T02:00:45.6595377Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6595648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6595779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6596070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6596210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6596215Z 2022-11-23T02:00:45.6596293Z Running tests... 2022-11-23T02:00:45.6596493Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6596759Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25976 2022-11-23T02:00:45.6596959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25977 2022-11-23T02:00:45.6597155Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6597444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6597570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6597860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6597993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6598169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6598444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6598570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6598855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6599031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6599206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6599515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6599818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6599987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6600206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6600265Z ok (3.745s) 2022-11-23T02:00:45.6600282Z 2022-11-23T02:00:45.6600649Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6600730Z Ran 1 test in 3.745s 2022-11-23T02:00:45.6600736Z 2022-11-23T02:00:45.6600800Z OK 2022-11-23T02:00:45.6600805Z 2022-11-23T02:00:45.6600895Z Generating XML reports... 2022-11-23T02:00:45.6601265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015617.xml 2022-11-23T02:00:45.6601518Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6601815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6601944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6602250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6602395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6602400Z 2022-11-23T02:00:45.6602475Z Running tests... 2022-11-23T02:00:45.6602690Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6602967Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26155 2022-11-23T02:00:45.6603134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26156 2022-11-23T02:00:45.6603343Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6603641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6603776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6604131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6604281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6604475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6609583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6609739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6610088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6610231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6610421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6610751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6611071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6611342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6611516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6611575Z ok (3.750s) 2022-11-23T02:00:45.6611590Z 2022-11-23T02:00:45.6611798Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6611875Z Ran 1 test in 3.750s 2022-11-23T02:00:45.6611880Z 2022-11-23T02:00:45.6611941Z OK 2022-11-23T02:00:45.6611946Z 2022-11-23T02:00:45.6612035Z Generating XML reports... 2022-11-23T02:00:45.6612516Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015624.xml 2022-11-23T02:00:45.6612760Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6613042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6613168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6613460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6613596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6613601Z 2022-11-23T02:00:45.6613672Z Running tests... 2022-11-23T02:00:45.6613872Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6614078Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:00:45.6614083Z 2022-11-23T02:00:45.6614281Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6614355Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6614360Z 2022-11-23T02:00:45.6614427Z OK (skipped=1) 2022-11-23T02:00:45.6614432Z 2022-11-23T02:00:45.6614512Z Generating XML reports... 2022-11-23T02:00:45.6614863Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015630.xml 2022-11-23T02:00:45.6615101Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6615380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6615500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6615782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6615913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6615962Z 2022-11-23T02:00:45.6616031Z Running tests... 2022-11-23T02:00:45.6616234Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6616457Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.001s) 2022-11-23T02:00:45.6616465Z 2022-11-23T02:00:45.6616663Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6616738Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6616743Z 2022-11-23T02:00:45.6616813Z OK (skipped=1) 2022-11-23T02:00:45.6616817Z 2022-11-23T02:00:45.6616904Z Generating XML reports... 2022-11-23T02:00:45.6617247Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015633.xml 2022-11-23T02:00:45.6617487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6617771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6617896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6618185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6618375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6618381Z 2022-11-23T02:00:45.6618452Z Running tests... 2022-11-23T02:00:45.6618651Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6619371Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.518s) 2022-11-23T02:00:45.6619379Z 2022-11-23T02:00:45.6619576Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6619649Z Ran 1 test in 0.519s 2022-11-23T02:00:45.6619654Z 2022-11-23T02:00:45.6619722Z OK (skipped=1) 2022-11-23T02:00:45.6619726Z 2022-11-23T02:00:45.6619807Z Generating XML reports... 2022-11-23T02:00:45.6620154Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015637.xml 2022-11-23T02:00:45.6620392Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6620671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6620795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6621075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6621213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6621218Z 2022-11-23T02:00:45.6621290Z Running tests... 2022-11-23T02:00:45.6621490Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6621744Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26508 2022-11-23T02:00:45.6621903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26509 2022-11-23T02:00:45.6622102Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6622389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6622514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6622806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6622981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6623160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6623444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6623566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6623851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6623984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6624163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6624467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6624772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6624935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6625128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6625320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo72lzjvz 2022-11-23T02:00:45.6625516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo72lzjvz/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6625690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1eouxljl 2022-11-23T02:00:45.6625883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1eouxljl/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6625951Z ok (6.761s) 2022-11-23T02:00:45.6625958Z 2022-11-23T02:00:45.6626167Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6626243Z Ran 1 test in 6.762s 2022-11-23T02:00:45.6626249Z 2022-11-23T02:00:45.6626309Z OK 2022-11-23T02:00:45.6626314Z 2022-11-23T02:00:45.6626399Z Generating XML reports... 2022-11-23T02:00:45.6626751Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015640.xml 2022-11-23T02:00:45.6626990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6627427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6627550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6627840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6627972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6627980Z 2022-11-23T02:00:45.6628053Z Running tests... 2022-11-23T02:00:45.6628259Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6628502Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26704 2022-11-23T02:00:45.6628657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26705 2022-11-23T02:00:45.6628855Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6629131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6629252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6629536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6629708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6629873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6630156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6630280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6630573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6630712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6630891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6631202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6631504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6631668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6631865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6631935Z ok (4.351s) 2022-11-23T02:00:45.6631940Z 2022-11-23T02:00:45.6632142Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6632216Z Ran 1 test in 4.351s 2022-11-23T02:00:45.6632221Z 2022-11-23T02:00:45.6632279Z OK 2022-11-23T02:00:45.6632283Z 2022-11-23T02:00:45.6632364Z Generating XML reports... 2022-11-23T02:00:45.6632711Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015650.xml 2022-11-23T02:00:45.6632948Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6633234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6633353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6633642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6633775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6633779Z 2022-11-23T02:00:45.6633848Z Running tests... 2022-11-23T02:00:45.6634040Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6634294Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26894 2022-11-23T02:00:45.6634449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26895 2022-11-23T02:00:45.6634647Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6634934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6635060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6635348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6635485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6635663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6635943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6636070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6636359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6636527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6636706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6637012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6637314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6637478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6637641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6637706Z ok (4.452s) 2022-11-23T02:00:45.6637711Z 2022-11-23T02:00:45.6637909Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6637982Z Ran 1 test in 4.452s 2022-11-23T02:00:45.6637987Z 2022-11-23T02:00:45.6638047Z OK 2022-11-23T02:00:45.6638052Z 2022-11-23T02:00:45.6638132Z Generating XML reports... 2022-11-23T02:00:45.6638470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015657.xml 2022-11-23T02:00:45.6638741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6639021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6639143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6639434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6639571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6639577Z 2022-11-23T02:00:45.6639652Z Running tests... 2022-11-23T02:00:45.6639852Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6641023Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.517s) 2022-11-23T02:00:45.6641030Z 2022-11-23T02:00:45.6641246Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6641326Z Ran 1 test in 0.517s 2022-11-23T02:00:45.6641331Z 2022-11-23T02:00:45.6641405Z OK (skipped=1) 2022-11-23T02:00:45.6641410Z 2022-11-23T02:00:45.6641497Z Generating XML reports... 2022-11-23T02:00:45.6641877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015705.xml 2022-11-23T02:00:45.6642139Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6642443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6642577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6642892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6643036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6643042Z 2022-11-23T02:00:45.6643119Z Running tests... 2022-11-23T02:00:45.6643335Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6644132Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.516s) 2022-11-23T02:00:45.6644213Z 2022-11-23T02:00:45.6644427Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6644503Z Ran 1 test in 0.516s 2022-11-23T02:00:45.6644510Z 2022-11-23T02:00:45.6644582Z OK (skipped=1) 2022-11-23T02:00:45.6644586Z 2022-11-23T02:00:45.6644672Z Generating XML reports... 2022-11-23T02:00:45.6645044Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015709.xml 2022-11-23T02:00:45.6645300Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6645597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6645730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6646050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6646195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6646200Z 2022-11-23T02:00:45.6646277Z Running tests... 2022-11-23T02:00:45.6646535Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6647361Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75052 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.517s) 2022-11-23T02:00:45.6647368Z 2022-11-23T02:00:45.6647583Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6647666Z Ran 1 test in 0.517s 2022-11-23T02:00:45.6647672Z 2022-11-23T02:00:45.6647747Z OK (skipped=1) 2022-11-23T02:00:45.6647752Z 2022-11-23T02:00:45.6647843Z Generating XML reports... 2022-11-23T02:00:45.6648219Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015712.xml 2022-11-23T02:00:45.6648478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6648786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6648921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6649233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6649378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6649384Z 2022-11-23T02:00:45.6649459Z Running tests... 2022-11-23T02:00:45.6649676Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6650515Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75139 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.516s) 2022-11-23T02:00:45.6650522Z 2022-11-23T02:00:45.6650738Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6650818Z Ran 1 test in 0.516s 2022-11-23T02:00:45.6650823Z 2022-11-23T02:00:45.6650897Z OK (skipped=1) 2022-11-23T02:00:45.6650902Z 2022-11-23T02:00:45.6650986Z Generating XML reports... 2022-11-23T02:00:45.6651472Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015716.xml 2022-11-23T02:00:45.6651764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6652063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6652192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6652601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6652738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6652743Z 2022-11-23T02:00:45.6652805Z Running tests... 2022-11-23T02:00:45.6653001Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6653721Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.517s) 2022-11-23T02:00:45.6653740Z 2022-11-23T02:00:45.6653929Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6654038Z Ran 1 test in 0.517s 2022-11-23T02:00:45.6654043Z 2022-11-23T02:00:45.6654114Z OK (skipped=1) 2022-11-23T02:00:45.6654118Z 2022-11-23T02:00:45.6654201Z Generating XML reports... 2022-11-23T02:00:45.6654549Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015719.xml 2022-11-23T02:00:45.6654787Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6655071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6655198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6655488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6655621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6655626Z 2022-11-23T02:00:45.6655693Z Running tests... 2022-11-23T02:00:45.6655892Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6656088Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6656093Z 2022-11-23T02:00:45.6656289Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6656362Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6656367Z 2022-11-23T02:00:45.6656439Z OK (skipped=1) 2022-11-23T02:00:45.6656443Z 2022-11-23T02:00:45.6656526Z Generating XML reports... 2022-11-23T02:00:45.6656870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015723.xml 2022-11-23T02:00:45.6657109Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6657387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6657510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6657798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6657922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6657936Z 2022-11-23T02:00:45.6657997Z Running tests... 2022-11-23T02:00:45.6658201Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6658397Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6658435Z 2022-11-23T02:00:45.6658634Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6658711Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6658716Z 2022-11-23T02:00:45.6658787Z OK (skipped=1) 2022-11-23T02:00:45.6658791Z 2022-11-23T02:00:45.6658875Z Generating XML reports... 2022-11-23T02:00:45.6659220Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015726.xml 2022-11-23T02:00:45.6659459Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6659739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6659865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6660154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6660288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6660295Z 2022-11-23T02:00:45.6660366Z Running tests... 2022-11-23T02:00:45.6660566Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6660799Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6660805Z 2022-11-23T02:00:45.6661006Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6661078Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6661083Z 2022-11-23T02:00:45.6661153Z OK (skipped=1) 2022-11-23T02:00:45.6661157Z 2022-11-23T02:00:45.6661239Z Generating XML reports... 2022-11-23T02:00:45.6661579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015729.xml 2022-11-23T02:00:45.6661820Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6662094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6662215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6662501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6662638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6662643Z 2022-11-23T02:00:45.6662718Z Running tests... 2022-11-23T02:00:45.6662923Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6663114Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6663119Z 2022-11-23T02:00:45.6663315Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6663389Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6663393Z 2022-11-23T02:00:45.6663467Z OK (skipped=1) 2022-11-23T02:00:45.6663472Z 2022-11-23T02:00:45.6663552Z Generating XML reports... 2022-11-23T02:00:45.6663897Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015732.xml 2022-11-23T02:00:45.6664134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6664417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6664541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6664829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6664963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6664968Z 2022-11-23T02:00:45.6665038Z Running tests... 2022-11-23T02:00:45.6665235Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6665465Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6665470Z 2022-11-23T02:00:45.6665666Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6665739Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6665743Z 2022-11-23T02:00:45.6665812Z OK (skipped=1) 2022-11-23T02:00:45.6665816Z 2022-11-23T02:00:45.6665889Z Generating XML reports... 2022-11-23T02:00:45.6666226Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015735.xml 2022-11-23T02:00:45.6666461Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6666738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6666861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6667150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6667283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6667288Z 2022-11-23T02:00:45.6667361Z Running tests... 2022-11-23T02:00:45.6667594Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6667786Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6667791Z 2022-11-23T02:00:45.6667987Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6668061Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6668065Z 2022-11-23T02:00:45.6668136Z OK (skipped=1) 2022-11-23T02:00:45.6668140Z 2022-11-23T02:00:45.6668220Z Generating XML reports... 2022-11-23T02:00:45.6668566Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015738.xml 2022-11-23T02:00:45.6668805Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6669084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6669210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6669494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6669629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6669633Z 2022-11-23T02:00:45.6669705Z Running tests... 2022-11-23T02:00:45.6669901Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6670085Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6670099Z 2022-11-23T02:00:45.6670284Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6670356Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6670361Z 2022-11-23T02:00:45.6670430Z OK (skipped=1) 2022-11-23T02:00:45.6670434Z 2022-11-23T02:00:45.6670516Z Generating XML reports... 2022-11-23T02:00:45.6670863Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015741.xml 2022-11-23T02:00:45.6671101Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6671383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6671507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6671796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6671931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6671985Z 2022-11-23T02:00:45.6672058Z Running tests... 2022-11-23T02:00:45.6672261Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6672454Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6672459Z 2022-11-23T02:00:45.6672659Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6672732Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6672736Z 2022-11-23T02:00:45.6672804Z OK (skipped=1) 2022-11-23T02:00:45.6672808Z 2022-11-23T02:00:45.6672892Z Generating XML reports... 2022-11-23T02:00:45.6673232Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015744.xml 2022-11-23T02:00:45.6673469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6673748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6673872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6674159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6674318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6674331Z 2022-11-23T02:00:45.6674391Z Running tests... 2022-11-23T02:00:45.6674593Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6674776Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.6674781Z 2022-11-23T02:00:45.6674979Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6675051Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6675056Z 2022-11-23T02:00:45.6675124Z OK (skipped=1) 2022-11-23T02:00:45.6675128Z 2022-11-23T02:00:45.6675213Z Generating XML reports... 2022-11-23T02:00:45.6675557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015747.xml 2022-11-23T02:00:45.6675796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6676078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6676202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6676490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6676626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6676631Z 2022-11-23T02:00:45.6676702Z Running tests... 2022-11-23T02:00:45.6676898Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6677088Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6677095Z 2022-11-23T02:00:45.6677289Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6677362Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6677367Z 2022-11-23T02:00:45.6677437Z OK (skipped=1) 2022-11-23T02:00:45.6677442Z 2022-11-23T02:00:45.6677525Z Generating XML reports... 2022-11-23T02:00:45.6677859Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015750.xml 2022-11-23T02:00:45.6678096Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6678369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6678491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6678776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6678949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6678954Z 2022-11-23T02:00:45.6679023Z Running tests... 2022-11-23T02:00:45.6679222Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6679453Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27954 2022-11-23T02:00:45.6679613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27955 2022-11-23T02:00:45.6679810Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6680089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6680268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6680739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6680884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6681076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6681415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6681547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6681848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6681988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6682172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6682499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6682823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6682997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6683160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6683435Z STAGE:2022-11-23 01:57:56 27954:27954 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6683699Z STAGE:2022-11-23 01:57:56 27955:27955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6683972Z STAGE:2022-11-23 01:57:56 27955:27955 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6684254Z STAGE:2022-11-23 01:57:56 27955:27955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6684442Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6684719Z STAGE:2022-11-23 01:57:56 27954:27954 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6685002Z STAGE:2022-11-23 01:57:56 27954:27954 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6685185Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6685447Z STAGE:2022-11-23 01:57:56 27955:27955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6685715Z STAGE:2022-11-23 01:57:56 27955:27955 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6685991Z STAGE:2022-11-23 01:57:56 27955:27955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6686254Z STAGE:2022-11-23 01:57:56 27954:27954 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6686565Z STAGE:2022-11-23 01:57:56 27954:27954 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6686844Z STAGE:2022-11-23 01:57:56 27954:27954 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6687487Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:00:45.6687567Z warnings.warn( 2022-11-23T02:00:45.6688188Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:00:45.6688269Z warnings.warn( 2022-11-23T02:00:45.6688341Z ok (4.575s) 2022-11-23T02:00:45.6688346Z 2022-11-23T02:00:45.6688560Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6688639Z Ran 1 test in 4.576s 2022-11-23T02:00:45.6688644Z 2022-11-23T02:00:45.6688705Z OK 2022-11-23T02:00:45.6688710Z 2022-11-23T02:00:45.6688798Z Generating XML reports... 2022-11-23T02:00:45.6689205Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015753.xml 2022-11-23T02:00:45.6689460Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6689748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6689876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6690183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6690329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6690334Z 2022-11-23T02:00:45.6690408Z Running tests... 2022-11-23T02:00:45.6690619Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6690827Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6690832Z 2022-11-23T02:00:45.6691039Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6691115Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6691120Z 2022-11-23T02:00:45.6691194Z OK (skipped=1) 2022-11-23T02:00:45.6691198Z 2022-11-23T02:00:45.6691286Z Generating XML reports... 2022-11-23T02:00:45.6691651Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015801.xml 2022-11-23T02:00:45.6691903Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6692204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6692336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6692641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6692894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6692899Z 2022-11-23T02:00:45.6692972Z Running tests... 2022-11-23T02:00:45.6693172Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6693420Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28204 2022-11-23T02:00:45.6693579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28205 2022-11-23T02:00:45.6693774Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6694093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6694207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6694496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6694628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6694804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6695083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6695207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6695495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6695629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6695805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6696142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6696445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6696608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6696772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6697032Z STAGE:2022-11-23 01:58:07 28205:28205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6697281Z STAGE:2022-11-23 01:58:07 28204:28204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6697542Z STAGE:2022-11-23 01:58:07 28205:28205 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6697812Z STAGE:2022-11-23 01:58:07 28205:28205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6697986Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6698247Z STAGE:2022-11-23 01:58:07 28204:28204 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6698512Z STAGE:2022-11-23 01:58:07 28204:28204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6698685Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6698937Z STAGE:2022-11-23 01:58:07 28204:28204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6699192Z STAGE:2022-11-23 01:58:07 28204:28204 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6699448Z STAGE:2022-11-23 01:58:07 28204:28204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6699698Z STAGE:2022-11-23 01:58:07 28205:28205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6699953Z STAGE:2022-11-23 01:58:07 28205:28205 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6700213Z STAGE:2022-11-23 01:58:07 28205:28205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6700279Z ok (4.455s) 2022-11-23T02:00:45.6700284Z 2022-11-23T02:00:45.6700488Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6700563Z Ran 1 test in 4.456s 2022-11-23T02:00:45.6700568Z 2022-11-23T02:00:45.6700626Z OK 2022-11-23T02:00:45.6700631Z 2022-11-23T02:00:45.6700713Z Generating XML reports... 2022-11-23T02:00:45.6701104Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015804.xml 2022-11-23T02:00:45.6701339Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6701622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6701744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6702034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6702168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6702173Z 2022-11-23T02:00:45.6702246Z Running tests... 2022-11-23T02:00:45.6702451Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6702690Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28396 2022-11-23T02:00:45.6702851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28397 2022-11-23T02:00:45.6703048Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6703365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6703488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6703769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6703906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6704083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6704363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6704489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6704775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6704912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6705086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6705390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6705693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6705860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6706021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6706086Z ok (4.354s) 2022-11-23T02:00:45.6706091Z 2022-11-23T02:00:45.6706293Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6706366Z Ran 1 test in 4.354s 2022-11-23T02:00:45.6706372Z 2022-11-23T02:00:45.6706430Z OK 2022-11-23T02:00:45.6706437Z 2022-11-23T02:00:45.6706522Z Generating XML reports... 2022-11-23T02:00:45.6706870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015812.xml 2022-11-23T02:00:45.6707109Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6707388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6707512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6707799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6707965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6707982Z 2022-11-23T02:00:45.6708042Z Running tests... 2022-11-23T02:00:45.6708242Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6708428Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6708433Z 2022-11-23T02:00:45.6708629Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6708703Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6708708Z 2022-11-23T02:00:45.6708775Z OK (skipped=1) 2022-11-23T02:00:45.6708779Z 2022-11-23T02:00:45.6708858Z Generating XML reports... 2022-11-23T02:00:45.6709204Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015819.xml 2022-11-23T02:00:45.6709442Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6709726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6709848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6710167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6710303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6710308Z 2022-11-23T02:00:45.6710381Z Running tests... 2022-11-23T02:00:45.6710583Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6710817Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28644 2022-11-23T02:00:45.6710975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28645 2022-11-23T02:00:45.6711173Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6711455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6711580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6711870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6712004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6712170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6712452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6712575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6712868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6713000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6713175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6713479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6713781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6713945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6714107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6714362Z STAGE:2022-11-23 01:58:25 28644:28644 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6714652Z STAGE:2022-11-23 01:58:25 28645:28645 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6714907Z STAGE:2022-11-23 01:58:25 28645:28645 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6715157Z STAGE:2022-11-23 01:58:25 28644:28644 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6715423Z STAGE:2022-11-23 01:58:25 28645:28645 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6715689Z STAGE:2022-11-23 01:58:25 28644:28644 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6715860Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6716032Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6716286Z STAGE:2022-11-23 01:58:25 28644:28644 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6716540Z STAGE:2022-11-23 01:58:25 28645:28645 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6716795Z STAGE:2022-11-23 01:58:25 28644:28644 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6717092Z STAGE:2022-11-23 01:58:25 28644:28644 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6717347Z STAGE:2022-11-23 01:58:25 28645:28645 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6717600Z STAGE:2022-11-23 01:58:25 28645:28645 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6717663Z ok (4.151s) 2022-11-23T02:00:45.6717669Z 2022-11-23T02:00:45.6717870Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6717944Z Ran 1 test in 4.151s 2022-11-23T02:00:45.6717949Z 2022-11-23T02:00:45.6718006Z OK 2022-11-23T02:00:45.6718011Z 2022-11-23T02:00:45.6718094Z Generating XML reports... 2022-11-23T02:00:45.6718442Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015822.xml 2022-11-23T02:00:45.6718680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6718962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6719084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6719371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6719505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6719510Z 2022-11-23T02:00:45.6719579Z Running tests... 2022-11-23T02:00:45.6719779Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6720017Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28836 2022-11-23T02:00:45.6720225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28837 2022-11-23T02:00:45.6720599Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6720895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6721024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6721327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6721468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6721656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6721989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6722118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6722599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6722743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6722930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6723255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6723575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6723746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6723920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6724192Z STAGE:2022-11-23 01:58:32 28836:28836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6724454Z STAGE:2022-11-23 01:58:32 28837:28837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6724779Z STAGE:2022-11-23 01:58:32 28836:28836 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6725063Z STAGE:2022-11-23 01:58:32 28836:28836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6725244Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6725511Z STAGE:2022-11-23 01:58:32 28837:28837 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6725787Z STAGE:2022-11-23 01:58:32 28837:28837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6725965Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6726232Z STAGE:2022-11-23 01:58:32 28836:28836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6726500Z STAGE:2022-11-23 01:58:32 28836:28836 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6726780Z STAGE:2022-11-23 01:58:32 28836:28836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6727046Z STAGE:2022-11-23 01:58:32 28837:28837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6727315Z STAGE:2022-11-23 01:58:32 28837:28837 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6727593Z STAGE:2022-11-23 01:58:32 28837:28837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6727652Z ok (4.253s) 2022-11-23T02:00:45.6727669Z 2022-11-23T02:00:45.6727876Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6727954Z Ran 1 test in 4.254s 2022-11-23T02:00:45.6727959Z 2022-11-23T02:00:45.6728022Z OK 2022-11-23T02:00:45.6728026Z 2022-11-23T02:00:45.6728115Z Generating XML reports... 2022-11-23T02:00:45.6728487Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015829.xml 2022-11-23T02:00:45.6728736Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6729031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6729162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6729463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6729603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6729641Z 2022-11-23T02:00:45.6729715Z Running tests... 2022-11-23T02:00:45.6729926Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6730126Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6730131Z 2022-11-23T02:00:45.6730338Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6730417Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6730422Z 2022-11-23T02:00:45.6730492Z OK (skipped=1) 2022-11-23T02:00:45.6730497Z 2022-11-23T02:00:45.6730583Z Generating XML reports... 2022-11-23T02:00:45.6730944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015837.xml 2022-11-23T02:00:45.6731194Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6731489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6731622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6731926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6732092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6732108Z 2022-11-23T02:00:45.6732173Z Running tests... 2022-11-23T02:00:45.6732488Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6732668Z test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.6732674Z 2022-11-23T02:00:45.6732869Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6732943Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6732948Z 2022-11-23T02:00:45.6733017Z OK (skipped=1) 2022-11-23T02:00:45.6733021Z 2022-11-23T02:00:45.6733105Z Generating XML reports... 2022-11-23T02:00:45.6733443Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015840.xml 2022-11-23T02:00:45.6733680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6733962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6734084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6734367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6734500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6734505Z 2022-11-23T02:00:45.6734576Z Running tests... 2022-11-23T02:00:45.6734777Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6734967Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:00:45.6734974Z 2022-11-23T02:00:45.6735165Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6735238Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6735243Z 2022-11-23T02:00:45.6735312Z OK (skipped=1) 2022-11-23T02:00:45.6735316Z 2022-11-23T02:00:45.6735399Z Generating XML reports... 2022-11-23T02:00:45.6735741Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015843.xml 2022-11-23T02:00:45.6735967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6736246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6736371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6736657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6736957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6736961Z 2022-11-23T02:00:45.6737034Z Running tests... 2022-11-23T02:00:45.6737238Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6737431Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.6737436Z 2022-11-23T02:00:45.6737630Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6737699Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6737704Z 2022-11-23T02:00:45.6737773Z OK (skipped=1) 2022-11-23T02:00:45.6737777Z 2022-11-23T02:00:45.6737861Z Generating XML reports... 2022-11-23T02:00:45.6738206Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015846.xml 2022-11-23T02:00:45.6738444Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6738722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6738843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6739172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6739309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6739314Z 2022-11-23T02:00:45.6739384Z Running tests... 2022-11-23T02:00:45.6739584Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6739813Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29260 2022-11-23T02:00:45.6739970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29261 2022-11-23T02:00:45.6740168Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6740437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6740560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6740852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6740990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6741166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6741445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6741573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6741863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6742002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6742177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6742485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6742791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6742954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6743119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6743378Z STAGE:2022-11-23 01:58:51 29261:29261 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6743662Z STAGE:2022-11-23 01:58:51 29260:29260 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6743917Z STAGE:2022-11-23 01:58:52 29260:29260 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6744181Z STAGE:2022-11-23 01:58:52 29260:29260 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6744350Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6744607Z STAGE:2022-11-23 01:58:52 29261:29261 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6744858Z STAGE:2022-11-23 01:58:52 29260:29260 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6745123Z STAGE:2022-11-23 01:58:52 29261:29261 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6745294Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6745544Z STAGE:2022-11-23 01:58:52 29260:29260 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6745807Z STAGE:2022-11-23 01:58:52 29260:29260 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6746088Z STAGE:2022-11-23 01:58:52 29261:29261 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6746343Z STAGE:2022-11-23 01:58:52 29261:29261 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6746602Z STAGE:2022-11-23 01:58:52 29261:29261 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6746665Z ok (4.250s) 2022-11-23T02:00:45.6746670Z 2022-11-23T02:00:45.6746871Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6746945Z Ran 1 test in 4.251s 2022-11-23T02:00:45.6746949Z 2022-11-23T02:00:45.6747008Z OK 2022-11-23T02:00:45.6747012Z 2022-11-23T02:00:45.6747091Z Generating XML reports... 2022-11-23T02:00:45.6747440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015849.xml 2022-11-23T02:00:45.6747677Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6747959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6748082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6748369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6748504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6748509Z 2022-11-23T02:00:45.6748581Z Running tests... 2022-11-23T02:00:45.6748786Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6749026Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29452 2022-11-23T02:00:45.6749184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29453 2022-11-23T02:00:45.6749382Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6749664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6749787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6750066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6750202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6750380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6750661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6750824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6751114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6751248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6751424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6751727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6752033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6752195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6752358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6752615Z STAGE:2022-11-23 01:58:59 29453:29453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6752863Z STAGE:2022-11-23 01:58:59 29452:29452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6753151Z STAGE:2022-11-23 01:58:59 29452:29452 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6753418Z STAGE:2022-11-23 01:58:59 29452:29452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6753592Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6753848Z STAGE:2022-11-23 01:58:59 29453:29453 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6754115Z STAGE:2022-11-23 01:58:59 29453:29453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6754291Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6754540Z STAGE:2022-11-23 01:58:59 29453:29453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6754798Z STAGE:2022-11-23 01:58:59 29453:29453 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6755058Z STAGE:2022-11-23 01:58:59 29453:29453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6755301Z STAGE:2022-11-23 01:58:59 29452:29452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6755555Z STAGE:2022-11-23 01:58:59 29452:29452 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6755815Z STAGE:2022-11-23 01:58:59 29452:29452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6755884Z ok (4.553s) 2022-11-23T02:00:45.6755889Z 2022-11-23T02:00:45.6756090Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6756164Z Ran 1 test in 4.553s 2022-11-23T02:00:45.6756169Z 2022-11-23T02:00:45.6756224Z OK 2022-11-23T02:00:45.6756229Z 2022-11-23T02:00:45.6756311Z Generating XML reports... 2022-11-23T02:00:45.6756661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015856.xml 2022-11-23T02:00:45.6756897Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6757178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6757300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6757587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6757721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6757761Z 2022-11-23T02:00:45.6757835Z Running tests... 2022-11-23T02:00:45.6758037Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6758231Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.6758236Z 2022-11-23T02:00:45.6758436Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6758511Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6758516Z 2022-11-23T02:00:45.6758586Z OK (skipped=1) 2022-11-23T02:00:45.6758591Z 2022-11-23T02:00:45.6758674Z Generating XML reports... 2022-11-23T02:00:45.6759022Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015904.xml 2022-11-23T02:00:45.6759252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6759533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6759659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6759949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6760116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6760122Z 2022-11-23T02:00:45.6760238Z Running tests... 2022-11-23T02:00:45.6760608Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6760809Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:00:45.6760814Z 2022-11-23T02:00:45.6761022Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6761099Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6761104Z 2022-11-23T02:00:45.6761176Z OK (skipped=1) 2022-11-23T02:00:45.6761182Z 2022-11-23T02:00:45.6761271Z Generating XML reports... 2022-11-23T02:00:45.6761634Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015907.xml 2022-11-23T02:00:45.6761880Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6762175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6762306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6762608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6762748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6762753Z 2022-11-23T02:00:45.6762826Z Running tests... 2022-11-23T02:00:45.6763039Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6763354Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:00:45.6763362Z 2022-11-23T02:00:45.6763571Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6763650Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6763655Z 2022-11-23T02:00:45.6763718Z OK (skipped=1) 2022-11-23T02:00:45.6763732Z 2022-11-23T02:00:45.6763813Z Generating XML reports... 2022-11-23T02:00:45.6764177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015910.xml 2022-11-23T02:00:45.6764429Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6764726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6764860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6765163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6765356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6765361Z 2022-11-23T02:00:45.6765435Z Running tests... 2022-11-23T02:00:45.6765646Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6765854Z test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:00:45.6765859Z 2022-11-23T02:00:45.6766067Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6766144Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6766149Z 2022-11-23T02:00:45.6766222Z OK (skipped=1) 2022-11-23T02:00:45.6766227Z 2022-11-23T02:00:45.6766310Z Generating XML reports... 2022-11-23T02:00:45.6766673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015913.xml 2022-11-23T02:00:45.6766920Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6767218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6767348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6767689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6767832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6767837Z 2022-11-23T02:00:45.6767912Z Running tests... 2022-11-23T02:00:45.6768113Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6768335Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.001s) 2022-11-23T02:00:45.6768350Z 2022-11-23T02:00:45.6768547Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6768628Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6768634Z 2022-11-23T02:00:45.6768708Z OK (skipped=1) 2022-11-23T02:00:45.6768713Z 2022-11-23T02:00:45.6768800Z Generating XML reports... 2022-11-23T02:00:45.6769168Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015916.xml 2022-11-23T02:00:45.6769421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6769719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6769848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6770150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6770288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6770293Z 2022-11-23T02:00:45.6770369Z Running tests... 2022-11-23T02:00:45.6770578Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6770815Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.001s) 2022-11-23T02:00:45.6770820Z 2022-11-23T02:00:45.6771027Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6771104Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6771109Z 2022-11-23T02:00:45.6771183Z OK (skipped=1) 2022-11-23T02:00:45.6771188Z 2022-11-23T02:00:45.6771273Z Generating XML reports... 2022-11-23T02:00:45.6771636Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015919.xml 2022-11-23T02:00:45.6771882Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6772287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6772447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6772727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6772867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6772881Z 2022-11-23T02:00:45.6772943Z Running tests... 2022-11-23T02:00:45.6773143Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6773365Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.001s) 2022-11-23T02:00:45.6773371Z 2022-11-23T02:00:45.6773569Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6773644Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6773649Z 2022-11-23T02:00:45.6773716Z OK (skipped=1) 2022-11-23T02:00:45.6773724Z 2022-11-23T02:00:45.6773809Z Generating XML reports... 2022-11-23T02:00:45.6774152Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015922.xml 2022-11-23T02:00:45.6774393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6774705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6774830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6775116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6775251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6775256Z 2022-11-23T02:00:45.6775324Z Running tests... 2022-11-23T02:00:45.6775521Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6775734Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:00:45.6775739Z 2022-11-23T02:00:45.6775935Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6776007Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6776012Z 2022-11-23T02:00:45.6776083Z OK (skipped=1) 2022-11-23T02:00:45.6776087Z 2022-11-23T02:00:45.6776169Z Generating XML reports... 2022-11-23T02:00:45.6776512Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015925.xml 2022-11-23T02:00:45.6776747Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6777015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6777137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6777422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6777558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6777563Z 2022-11-23T02:00:45.6777632Z Running tests... 2022-11-23T02:00:45.6777836Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6778060Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30108 2022-11-23T02:00:45.6778220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30109 2022-11-23T02:00:45.6778415Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6778692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6778814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6779156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6779287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6779468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6779746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6779868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6780158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6780294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6780474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6780781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6781084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6781307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6781475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6781531Z ok (4.668s) 2022-11-23T02:00:45.6781536Z 2022-11-23T02:00:45.6781741Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6781818Z Ran 1 test in 4.669s 2022-11-23T02:00:45.6781822Z 2022-11-23T02:00:45.6781882Z OK 2022-11-23T02:00:45.6781887Z 2022-11-23T02:00:45.6781972Z Generating XML reports... 2022-11-23T02:00:45.6782316Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015928.xml 2022-11-23T02:00:45.6782555Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6782835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6782957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6783246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6783382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6783386Z 2022-11-23T02:00:45.6783455Z Running tests... 2022-11-23T02:00:45.6783656Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6783901Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30294 2022-11-23T02:00:45.6784056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30295 2022-11-23T02:00:45.6784251Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6784538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6784661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6784951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6785086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6785262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6785543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6785691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6785978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6786112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6786290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6786598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6786903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6787067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6787233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6787493Z STAGE:2022-11-23 01:59:38 30295:30295 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6787741Z STAGE:2022-11-23 01:59:39 30294:30294 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6788028Z STAGE:2022-11-23 01:59:39 30294:30294 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6788300Z STAGE:2022-11-23 01:59:39 30294:30294 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6788469Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6788727Z STAGE:2022-11-23 01:59:39 30295:30295 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6788992Z STAGE:2022-11-23 01:59:39 30295:30295 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6789161Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 2 (function operator()) 2022-11-23T02:00:45.6789230Z ok (4.667s) 2022-11-23T02:00:45.6789235Z 2022-11-23T02:00:45.6789436Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6789512Z Ran 1 test in 4.668s 2022-11-23T02:00:45.6789517Z 2022-11-23T02:00:45.6789575Z OK 2022-11-23T02:00:45.6789580Z 2022-11-23T02:00:45.6789664Z Generating XML reports... 2022-11-23T02:00:45.6790013Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015936.xml 2022-11-23T02:00:45.6790242Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6790526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6790651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6790938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6791076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6791081Z 2022-11-23T02:00:45.6791153Z Running tests... 2022-11-23T02:00:45.6791354Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6791605Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30486 2022-11-23T02:00:45.6791761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30487 2022-11-23T02:00:45.6791953Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6792231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6792352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6792638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6792807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6792984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6793262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6793383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6793672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6793810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6793987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6794293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6794592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6794753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6794944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6795204Z STAGE:2022-11-23 01:59:46 30487:30487 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6795452Z STAGE:2022-11-23 01:59:46 30486:30486 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:00:45.6795706Z STAGE:2022-11-23 01:59:47 30486:30486 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6795982Z STAGE:2022-11-23 01:59:47 30486:30486 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6796157Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6796413Z STAGE:2022-11-23 01:59:47 30487:30487 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:00:45.6796677Z STAGE:2022-11-23 01:59:47 30487:30487 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:00:45.6796849Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:00:45.6796916Z ok (4.767s) 2022-11-23T02:00:45.6796921Z 2022-11-23T02:00:45.6797121Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6797196Z Ran 1 test in 4.768s 2022-11-23T02:00:45.6797201Z 2022-11-23T02:00:45.6797261Z OK 2022-11-23T02:00:45.6797266Z 2022-11-23T02:00:45.6797352Z Generating XML reports... 2022-11-23T02:00:45.6797701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015944.xml 2022-11-23T02:00:45.6797941Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6798212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6798335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6798625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6798759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6798764Z 2022-11-23T02:00:45.6798835Z Running tests... 2022-11-23T02:00:45.6799037Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6799244Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:00:45.6799250Z 2022-11-23T02:00:45.6799447Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6799558Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6799563Z 2022-11-23T02:00:45.6799633Z OK (skipped=1) 2022-11-23T02:00:45.6799638Z 2022-11-23T02:00:45.6799716Z Generating XML reports... 2022-11-23T02:00:45.6800065Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015951.xml 2022-11-23T02:00:45.6800519Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6800820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6800950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6801256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6801402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6801409Z 2022-11-23T02:00:45.6801487Z Running tests... 2022-11-23T02:00:45.6801701Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6801918Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:00:45.6801923Z 2022-11-23T02:00:45.6802179Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6802259Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6802264Z 2022-11-23T02:00:45.6802336Z OK (skipped=1) 2022-11-23T02:00:45.6802340Z 2022-11-23T02:00:45.6802417Z Generating XML reports... 2022-11-23T02:00:45.6802786Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015954.xml 2022-11-23T02:00:45.6803038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6803334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6803466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6803772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6803913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6803919Z 2022-11-23T02:00:45.6803993Z Running tests... 2022-11-23T02:00:45.6804202Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6804434Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:00:45.6804440Z 2022-11-23T02:00:45.6804645Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6804719Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6804724Z 2022-11-23T02:00:45.6804794Z OK (skipped=1) 2022-11-23T02:00:45.6804801Z 2022-11-23T02:00:45.6804887Z Generating XML reports... 2022-11-23T02:00:45.6805246Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015958.xml 2022-11-23T02:00:45.6805498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6805797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6805930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6806236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6806379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6806384Z 2022-11-23T02:00:45.6806462Z Running tests... 2022-11-23T02:00:45.6806676Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6806942Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s) 2022-11-23T02:00:45.6806948Z 2022-11-23T02:00:45.6807161Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6807229Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6807234Z 2022-11-23T02:00:45.6807314Z OK (skipped=1) 2022-11-23T02:00:45.6807320Z 2022-11-23T02:00:45.6807408Z Generating XML reports... 2022-11-23T02:00:45.6807767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020000.xml 2022-11-23T02:00:45.6808017Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6808311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6808437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6808739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6808879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6808884Z 2022-11-23T02:00:45.6808957Z Running tests... 2022-11-23T02:00:45.6809206Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6809425Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.001s) 2022-11-23T02:00:45.6809430Z 2022-11-23T02:00:45.6809640Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6809718Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6809723Z 2022-11-23T02:00:45.6809794Z OK (skipped=1) 2022-11-23T02:00:45.6809798Z 2022-11-23T02:00:45.6809884Z Generating XML reports... 2022-11-23T02:00:45.6810249Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020004.xml 2022-11-23T02:00:45.6810503Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6810801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6810935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6811238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6811379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6811385Z 2022-11-23T02:00:45.6811449Z Running tests... 2022-11-23T02:00:45.6811660Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6811885Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.001s) 2022-11-23T02:00:45.6811890Z 2022-11-23T02:00:45.6812100Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6812179Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6812184Z 2022-11-23T02:00:45.6812259Z OK (skipped=1) 2022-11-23T02:00:45.6812263Z 2022-11-23T02:00:45.6812454Z Generating XML reports... 2022-11-23T02:00:45.6812800Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020007.xml 2022-11-23T02:00:45.6813039Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6813318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6813438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6813724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6813858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6813900Z 2022-11-23T02:00:45.6813971Z Running tests... 2022-11-23T02:00:45.6814173Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6814410Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31026 2022-11-23T02:00:45.6814568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31027 2022-11-23T02:00:45.6814765Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6815046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6815171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6815460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6815599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6815775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6816081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6816208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6816499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6816634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6816813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6817119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6817422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6817589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6817754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6817822Z ok (6.571s) 2022-11-23T02:00:45.6817827Z 2022-11-23T02:00:45.6818026Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6818097Z Ran 1 test in 6.572s 2022-11-23T02:00:45.6818102Z 2022-11-23T02:00:45.6818158Z OK 2022-11-23T02:00:45.6818162Z 2022-11-23T02:00:45.6818243Z Generating XML reports... 2022-11-23T02:00:45.6818590Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020010.xml 2022-11-23T02:00:45.6818827Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6819106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6819225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6819518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6819654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6819659Z 2022-11-23T02:00:45.6819733Z Running tests... 2022-11-23T02:00:45.6819932Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6820125Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s) 2022-11-23T02:00:45.6820141Z 2022-11-23T02:00:45.6820330Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6820402Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6820406Z 2022-11-23T02:00:45.6820517Z OK (skipped=1) 2022-11-23T02:00:45.6820522Z 2022-11-23T02:00:45.6820606Z Generating XML reports... 2022-11-23T02:00:45.6820950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020019.xml 2022-11-23T02:00:45.6821188Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6821467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6821591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6821878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6822012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6822017Z 2022-11-23T02:00:45.6822089Z Running tests... 2022-11-23T02:00:45.6822285Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6822515Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31280 2022-11-23T02:00:45.6822669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31281 2022-11-23T02:00:45.6822896Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6823323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6823448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6823732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6823866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6824041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6824323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6824449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6824731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6824868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6825045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6825350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6825653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6825817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6825981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6826166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpwapvp4 2022-11-23T02:00:45.6826361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpwapvp4/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6826546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ze9cppy 2022-11-23T02:00:45.6826738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ze9cppy/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6826803Z ok (4.354s) 2022-11-23T02:00:45.6826809Z 2022-11-23T02:00:45.6827006Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6827077Z Ran 1 test in 4.355s 2022-11-23T02:00:45.6827081Z 2022-11-23T02:00:45.6827143Z OK 2022-11-23T02:00:45.6827147Z 2022-11-23T02:00:45.6827261Z Generating XML reports... 2022-11-23T02:00:45.6827606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020022.xml 2022-11-23T02:00:45.6827843Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6828127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6828250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6828539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6828672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6828677Z 2022-11-23T02:00:45.6828738Z Running tests... 2022-11-23T02:00:45.6828943Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6829200Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31466 2022-11-23T02:00:45.6829361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31467 2022-11-23T02:00:45.6829605Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:00:45.6829885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6830009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6830299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6830436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6830610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:00:45.6830891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6831011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6831301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6831434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6831607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:00:45.6831913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6832218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:00:45.6832383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:00:45.6832550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:00:45.6832736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmg4k17lu 2022-11-23T02:00:45.6832936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmg4k17lu/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6833119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshdpjqf4 2022-11-23T02:00:45.6833314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshdpjqf4/_remote_module_non_scriptable.py 2022-11-23T02:00:45.6833900Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:00:45.6834530Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:00:45.6834597Z ok (6.272s) 2022-11-23T02:00:45.6834602Z 2022-11-23T02:00:45.6834806Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6834879Z Ran 1 test in 6.272s 2022-11-23T02:00:45.6834884Z 2022-11-23T02:00:45.6834943Z OK 2022-11-23T02:00:45.6834948Z 2022-11-23T02:00:45.6835028Z Generating XML reports... 2022-11-23T02:00:45.6835374Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020030.xml 2022-11-23T02:00:45.6835610Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6835916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6836042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6836331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6836465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6836470Z 2022-11-23T02:00:45.6836541Z Running tests... 2022-11-23T02:00:45.6836739Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6837095Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:00:45.6837100Z 2022-11-23T02:00:45.6837296Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6837370Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6837374Z 2022-11-23T02:00:45.6837443Z OK (skipped=1) 2022-11-23T02:00:45.6837448Z 2022-11-23T02:00:45.6837530Z Generating XML reports... 2022-11-23T02:00:45.6837870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020039.xml 2022-11-23T02:00:45.6838107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:00:45.6838389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:00:45.6838510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:00:45.6838799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:00:45.6838934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:00:45.6838939Z 2022-11-23T02:00:45.6839008Z Running tests... 2022-11-23T02:00:45.6839208Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6839565Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:00:45.6839570Z 2022-11-23T02:00:45.6839766Z ---------------------------------------------------------------------- 2022-11-23T02:00:45.6839840Z Ran 1 test in 0.002s 2022-11-23T02:00:45.6839845Z 2022-11-23T02:00:45.6839912Z OK (skipped=1) 2022-11-23T02:00:45.6839917Z 2022-11-23T02:00:45.6839997Z Generating XML reports... 2022-11-23T02:00:45.6840545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020042.xml 2022-11-23T02:00:45.6840603Z 2022-11-23T02:00:45.6841012Z ##[endgroup] 2022-11-23T02:00:45.6841402Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_7ddbvpx2) 2022-11-23T02:00:45.6841408Z 2022-11-23T02:00:45.6841567Z Running distributed tests for the nccl backend with file init_method in shard 2 of 2 2022-11-23T02:00:45.6842041Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:00:45.500814] 2022-11-23T02:22:33.2908434Z 2022-11-23T02:22:33.2913253Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T02:22:33.2916776Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_6u__3oec) 2022-11-23T02:22:33.2927044Z 2022-11-23T02:22:33.3065533Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-11-23T02:22:33.3118918Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3119473Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3119942Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3120446Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3120926Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3121428Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3121953Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3122468Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3123022Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3123609Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3124223Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3124967Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3125538Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3126106Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3126602Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3127118Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3127618Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3128080Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3128549Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3129025Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3129553Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3130038Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3130474Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3130896Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3131357Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3131813Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3132251Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3132702Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3133126Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3133528Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3133946Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3134369Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3134785Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3135208Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3135644Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3136085Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3136508Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3136960Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3137461Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3137879Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3138327Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3138779Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3139262Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3139729Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3140194Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3140641Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3141100Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3141557Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3141982Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3142456Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3142918Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3143427Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3143860Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3144321Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3144753Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3145187Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3145627Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3146060Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3146488Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3146907Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3147351Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3147755Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3148165Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3148559Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3148975Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3149400Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3149822Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3150226Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3150626Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3151037Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3151453Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3151860Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3152291Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3152711Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3153126Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3153516Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3153922Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3154327Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3154761Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3155164Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3155572Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3156035Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3156456Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3156916Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3157371Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3157821Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3158302Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3158775Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3159254Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3159718Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3160265Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3160739Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3161188Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3161663Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3162208Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3162702Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3163176Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3163656Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3164099Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3164495Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3164908Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3165289Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3165672Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3166087Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3166508Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3166927Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3167313Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3167732Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3168167Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3168582Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3169093Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3169518Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3169968Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3170402Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3170851Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3171289Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3171737Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3172190Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3172642Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3173059Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3173461Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3173860Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3174267Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3174735Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3175143Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3175542Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3176045Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3176755Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3177229Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3177661Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3178107Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3178559Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3179028Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3179523Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3179992Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3180480Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3180953Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3181380Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3181768Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3182181Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3182613Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3183055Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3183515Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3183999Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3184433Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3184862Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3185350Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3185890Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3186478Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3187125Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3187763Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3188391Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3189051Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3189691Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3190304Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3190945Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3191522Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3192077Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3192549Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3192953Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3193381Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3193804Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3194204Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3194657Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3195117Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3195607Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3196097Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3196550Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3196944Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3197384Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3197879Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3198335Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3198777Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3199205Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3199665Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3200098Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3200614Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3201044Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3201475Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3201915Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3202339Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3202788Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3203257Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3203725Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3204144Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3204549Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3204961Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3205412Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3205842Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3206237Z test_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3206626Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3207013Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3207417Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3207810Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3208182Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3208601Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3208997Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3209382Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3209749Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3210151Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3210612Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3211009Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3211415Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3211777Z test_isend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3212183Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3212608Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3213044Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3213512Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3213985Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3214425Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3214873Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3215335Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3215792Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3216236Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3216718Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3217152Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3217598Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3218029Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3218452Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3218871Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3219351Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3219881Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3220383Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3220858Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3221341Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3221830Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3222286Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3222723Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3223171Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3223643Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3224116Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3224633Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3225173Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3225692Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3226119Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3226539Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3226960Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3227384Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3227789Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3228184Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3228642Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3229031Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3229419Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3229796Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3230190Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3230592Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3231011Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3231424Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3231818Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3232204Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3232610Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3233013Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3263099Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3263535Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3263939Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3264322Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3264901Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3265318Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3265717Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3266123Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3266514Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3266894Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3267342Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3267816Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3268281Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3268696Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3269123Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3269576Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3269996Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3270410Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3270843Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3271298Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3271731Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3272151Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3272568Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3272986Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3273383Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3273828Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3274306Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3274766Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3275620Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3276311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3276795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3277429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3278011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3278249Z 2022-11-23T02:22:33.3278351Z Running tests... 2022-11-23T02:22:33.3278802Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3279463Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31836 2022-11-23T02:22:33.3280100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31837 2022-11-23T02:22:33.3280712Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3281425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3281901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3282528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3283019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3283558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3284247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3284735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3285358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3285846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3286318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3287041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3287791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3288345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3288843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3289387Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:22:33.3290305Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:22:33.3291016Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:22:33.3291926Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:22:33.3292632Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:22:33.3293525Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:22:33.3294223Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:22:33.3295122Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:22:33.3295823Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:22:33.3296759Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:22:33.3297462Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:22:33.3298353Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:22:33.3299038Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:22:33.3299925Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:22:33.3300624Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:22:33.3301523Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:22:33.3302027Z ok (4.557s) 2022-11-23T02:22:33.3302173Z 2022-11-23T02:22:33.3302509Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3302837Z Ran 1 test in 4.557s 2022-11-23T02:22:33.3302998Z 2022-11-23T02:22:33.3303083Z OK 2022-11-23T02:22:33.3303213Z 2022-11-23T02:22:33.3303329Z Generating XML reports... 2022-11-23T02:22:33.3303999Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020048.xml 2022-11-23T02:22:33.3304715Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3305388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3305864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3306476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3306972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3307210Z 2022-11-23T02:22:33.3307311Z Running tests... 2022-11-23T02:22:33.3307746Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3308280Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s) 2022-11-23T02:22:33.3308600Z 2022-11-23T02:22:33.3308886Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3309223Z Ran 1 test in 0.003s 2022-11-23T02:22:33.3309382Z 2022-11-23T02:22:33.3309467Z OK (skipped=1) 2022-11-23T02:22:33.3309618Z 2022-11-23T02:22:33.3309737Z Generating XML reports... 2022-11-23T02:22:33.3310404Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020056.xml 2022-11-23T02:22:33.3311113Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3311783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3312255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3312883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3313377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3313593Z 2022-11-23T02:22:33.3313694Z Running tests... 2022-11-23T02:22:33.3314129Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3314703Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32084 2022-11-23T02:22:33.3315344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32085 2022-11-23T02:22:33.3315878Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3316592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3317066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3317677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3318169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3318643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3319323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3319793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3320480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3321024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3321491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3322215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3322968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3323521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3324016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3324376Z ok (3.739s) 2022-11-23T02:22:33.3324521Z 2022-11-23T02:22:33.3324816Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3325154Z Ran 1 test in 3.740s 2022-11-23T02:22:33.3325298Z 2022-11-23T02:22:33.3325383Z OK 2022-11-23T02:22:33.3325515Z 2022-11-23T02:22:33.3325633Z Generating XML reports... 2022-11-23T02:22:33.3326294Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020059.xml 2022-11-23T02:22:33.3327008Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3327677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3328149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3328770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3329249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3329478Z 2022-11-23T02:22:33.3329579Z Running tests... 2022-11-23T02:22:33.3330015Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3331277Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.513s) 2022-11-23T02:22:33.3331948Z 2022-11-23T02:22:33.3332233Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3332572Z Ran 1 test in 0.513s 2022-11-23T02:22:33.3332727Z 2022-11-23T02:22:33.3332883Z OK (skipped=1) 2022-11-23T02:22:33.3333033Z 2022-11-23T02:22:33.3333149Z Generating XML reports... 2022-11-23T02:22:33.3333810Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020106.xml 2022-11-23T02:22:33.3334505Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3335182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3335653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3336271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3336767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3337001Z 2022-11-23T02:22:33.3337102Z Running tests... 2022-11-23T02:22:33.3337537Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3338080Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.001s) 2022-11-23T02:22:33.3338424Z 2022-11-23T02:22:33.3338705Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3339040Z Ran 1 test in 0.001s 2022-11-23T02:22:33.3339243Z 2022-11-23T02:22:33.3339346Z OK (skipped=1) 2022-11-23T02:22:33.3339498Z 2022-11-23T02:22:33.3339614Z Generating XML reports... 2022-11-23T02:22:33.3340276Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020109.xml 2022-11-23T02:22:33.3340984Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3341651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3342108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3342738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3343228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3343460Z 2022-11-23T02:22:33.3343561Z Running tests... 2022-11-23T02:22:33.3343995Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3344567Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.001s) 2022-11-23T02:22:33.3344925Z 2022-11-23T02:22:33.3345213Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3345535Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3345691Z 2022-11-23T02:22:33.3345791Z OK (skipped=1) 2022-11-23T02:22:33.3345944Z 2022-11-23T02:22:33.3346062Z Generating XML reports... 2022-11-23T02:22:33.3346723Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020112.xml 2022-11-23T02:22:33.3347435Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3348105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3348578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3349188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3349681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3349910Z 2022-11-23T02:22:33.3350010Z Running tests... 2022-11-23T02:22:33.3350444Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3351058Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32437 2022-11-23T02:22:33.3351732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32438 2022-11-23T02:22:33.3352260Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3352968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3353423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3354043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3354537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3355025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3355714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3356191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3356821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3357296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3357816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3358532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3359276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3359816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3360412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3360945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt0osfp6 2022-11-23T02:22:33.3361524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt0osfp6/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3362075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1m2pgsma 2022-11-23T02:22:33.3362632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1m2pgsma/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3363641Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3364201Z warnings.warn( 2022-11-23T02:22:33.3364577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3365517Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3366074Z warnings.warn( 2022-11-23T02:22:33.3366449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3366940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3367425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3367910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3368386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3368870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3369348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3369884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3370356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3370836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3371303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3371640Z ok (6.847s) 2022-11-23T02:22:33.3371775Z 2022-11-23T02:22:33.3372063Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3372388Z Ran 1 test in 6.847s 2022-11-23T02:22:33.3372539Z 2022-11-23T02:22:33.3372614Z OK 2022-11-23T02:22:33.3372728Z 2022-11-23T02:22:33.3372838Z Generating XML reports... 2022-11-23T02:22:33.3373490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020115.xml 2022-11-23T02:22:33.3374192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3374844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3375301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3375959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3376440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3376674Z 2022-11-23T02:22:33.3376759Z Running tests... 2022-11-23T02:22:33.3377199Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3377823Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32633 2022-11-23T02:22:33.3378441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32634 2022-11-23T02:22:33.3378968Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3379658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3380118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3380724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3381191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3381651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3382318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3382771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3383386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3383851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3384308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3384998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3385736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3386280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3386755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3387268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0vz55e_ 2022-11-23T02:22:33.3387874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0vz55e_/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3388417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmcy6alv 2022-11-23T02:22:33.3388964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmcy6alv/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3389516Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3390072Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3391034Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3391597Z warnings.warn( 2022-11-23T02:22:33.3391974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3392957Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3393515Z warnings.warn( 2022-11-23T02:22:33.3393894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3394377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3394861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3395202Z ok (4.543s) 2022-11-23T02:22:33.3395337Z 2022-11-23T02:22:33.3395627Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3395950Z Ran 1 test in 4.544s 2022-11-23T02:22:33.3396098Z 2022-11-23T02:22:33.3396173Z OK 2022-11-23T02:22:33.3396292Z 2022-11-23T02:22:33.3396400Z Generating XML reports... 2022-11-23T02:22:33.3397040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020125.xml 2022-11-23T02:22:33.3397741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3398395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3398849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3399456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3399936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3400214Z 2022-11-23T02:22:33.3400305Z Running tests... 2022-11-23T02:22:33.3400723Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3401349Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32827 2022-11-23T02:22:33.3401982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32828 2022-11-23T02:22:33.3402505Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3403197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3403657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3404261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3404739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3405250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3405913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3406373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3406987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3407470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3407933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3408636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3409369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3409911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3410390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3410958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi388j1c_ 2022-11-23T02:22:33.3411514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi388j1c_/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3412064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegt7_x2u 2022-11-23T02:22:33.3412610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegt7_x2u/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3413591Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3414142Z warnings.warn( 2022-11-23T02:22:33.3414519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3415469Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3416033Z warnings.warn( 2022-11-23T02:22:33.3416416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3416913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3417414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3417903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3418383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3418871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3419351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3419695Z ok (4.643s) 2022-11-23T02:22:33.3419836Z 2022-11-23T02:22:33.3420129Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3420455Z Ran 1 test in 4.644s 2022-11-23T02:22:33.3420610Z 2022-11-23T02:22:33.3420678Z OK 2022-11-23T02:22:33.3420803Z 2022-11-23T02:22:33.3420921Z Generating XML reports... 2022-11-23T02:22:33.3421576Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020133.xml 2022-11-23T02:22:33.3422284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3422941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3423546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3424164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3424642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3424859Z 2022-11-23T02:22:33.3424958Z Running tests... 2022-11-23T02:22:33.3425383Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3426038Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33021 2022-11-23T02:22:33.3426689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33022 2022-11-23T02:22:33.3427208Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3427912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3428378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3429038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3429523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3429992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3430666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3431121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3431734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3432216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3432679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3433378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3434117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3434657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3435144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3435656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k24ir1d 2022-11-23T02:22:33.3436215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k24ir1d/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3436772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8a7hpmy 2022-11-23T02:22:33.3437319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8a7hpmy/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3438307Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3438872Z warnings.warn( 2022-11-23T02:22:33.3439248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3440266Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3440823Z warnings.warn( 2022-11-23T02:22:33.3441259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3441612Z ok (4.650s) 2022-11-23T02:22:33.3441751Z 2022-11-23T02:22:33.3442033Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3442369Z Ran 1 test in 4.651s 2022-11-23T02:22:33.3442522Z 2022-11-23T02:22:33.3442603Z OK 2022-11-23T02:22:33.3442728Z 2022-11-23T02:22:33.3442844Z Generating XML reports... 2022-11-23T02:22:33.3443507Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020140.xml 2022-11-23T02:22:33.3444210Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3444871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3445324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3445943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3446424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3446656Z 2022-11-23T02:22:33.3446757Z Running tests... 2022-11-23T02:22:33.3447258Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3447917Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33215 2022-11-23T02:22:33.3448560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33216 2022-11-23T02:22:33.3449082Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3449768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3450234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3450847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3451332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3451800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3452472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3452931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3453529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3454006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3454467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3455170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3455910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3456459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3456934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3457458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkph8lv9m 2022-11-23T02:22:33.3458010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkph8lv9m/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3458571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubpnly2t 2022-11-23T02:22:33.3459164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubpnly2t/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3460152Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3460720Z warnings.warn( 2022-11-23T02:22:33.3461109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3462057Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3462620Z warnings.warn( 2022-11-23T02:22:33.3463007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3463504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3463992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3464345Z ok (6.662s) 2022-11-23T02:22:33.3464486Z 2022-11-23T02:22:33.3464772Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3465150Z Ran 1 test in 6.662s 2022-11-23T02:22:33.3465308Z 2022-11-23T02:22:33.3465387Z OK 2022-11-23T02:22:33.3465507Z 2022-11-23T02:22:33.3465623Z Generating XML reports... 2022-11-23T02:22:33.3466283Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020148.xml 2022-11-23T02:22:33.3466979Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3467645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3468109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3468726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3469214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3469437Z 2022-11-23T02:22:33.3469538Z Running tests... 2022-11-23T02:22:33.3469972Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3470586Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33411 2022-11-23T02:22:33.3471197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33412 2022-11-23T02:22:33.3471724Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3472416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3472878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3473495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3473985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3474449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3475113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3475570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3476179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3476655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3477177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3477875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3478625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3479158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3479640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3480197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwl_d0rj 2022-11-23T02:22:33.3480756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwl_d0rj/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3481315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc_8e5mo 2022-11-23T02:22:33.3481874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc_8e5mo/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3482913Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3483486Z warnings.warn( 2022-11-23T02:22:33.3483864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3484809Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3485375Z warnings.warn( 2022-11-23T02:22:33.3485750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3486247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3486750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3487096Z ok (6.952s) 2022-11-23T02:22:33.3487240Z 2022-11-23T02:22:33.3487533Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3487858Z Ran 1 test in 6.952s 2022-11-23T02:22:33.3488008Z 2022-11-23T02:22:33.3488087Z OK 2022-11-23T02:22:33.3488214Z 2022-11-23T02:22:33.3488326Z Generating XML reports... 2022-11-23T02:22:33.3488982Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020158.xml 2022-11-23T02:22:33.3489687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3490352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3490817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3491426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3491908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3492136Z 2022-11-23T02:22:33.3492233Z Running tests... 2022-11-23T02:22:33.3492662Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3493313Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33607 2022-11-23T02:22:33.3493957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33608 2022-11-23T02:22:33.3494485Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3495165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3495691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3496307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3496791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3497264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3497940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3498395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3499010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3499486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3499952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3500656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3501468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3502022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3502517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3503031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqqhaziq 2022-11-23T02:22:33.3503584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqqhaziq/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3504145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw50y_qg2 2022-11-23T02:22:33.3504702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw50y_qg2/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3505275Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3505837Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3506815Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3507376Z warnings.warn( 2022-11-23T02:22:33.3507760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3508707Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.3509256Z warnings.warn( 2022-11-23T02:22:33.3509638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3510141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3510632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3510991Z ok (4.744s) 2022-11-23T02:22:33.3511127Z 2022-11-23T02:22:33.3511419Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3511756Z Ran 1 test in 4.744s 2022-11-23T02:22:33.3511903Z 2022-11-23T02:22:33.3511979Z OK 2022-11-23T02:22:33.3512103Z 2022-11-23T02:22:33.3512217Z Generating XML reports... 2022-11-23T02:22:33.3512878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020208.xml 2022-11-23T02:22:33.3513644Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3514309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3514776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3515393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3515870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3516097Z 2022-11-23T02:22:33.3516195Z Running tests... 2022-11-23T02:22:33.3516618Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3517913Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.521s) 2022-11-23T02:22:33.3518583Z 2022-11-23T02:22:33.3518858Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3519229Z Ran 1 test in 0.521s 2022-11-23T02:22:33.3519386Z 2022-11-23T02:22:33.3519482Z OK (skipped=1) 2022-11-23T02:22:33.3519626Z 2022-11-23T02:22:33.3519742Z Generating XML reports... 2022-11-23T02:22:33.3520484Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020215.xml 2022-11-23T02:22:33.3521203Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3521873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3522345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3522968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3523450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3523682Z 2022-11-23T02:22:33.3523783Z Running tests... 2022-11-23T02:22:33.3524200Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3524802Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33859 2022-11-23T02:22:33.3525410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33860 2022-11-23T02:22:33.3525931Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3526627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3527094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3527706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3528194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3528653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3529326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3529792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3530406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3530885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3531413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3532118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3532865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3533399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3533884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3534238Z ok (3.739s) 2022-11-23T02:22:33.3534383Z 2022-11-23T02:22:33.3534672Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3535000Z Ran 1 test in 3.739s 2022-11-23T02:22:33.3535159Z 2022-11-23T02:22:33.3535239Z OK 2022-11-23T02:22:33.3535366Z 2022-11-23T02:22:33.3535470Z Generating XML reports... 2022-11-23T02:22:33.3536124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020219.xml 2022-11-23T02:22:33.3536823Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3537537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3538002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3538621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3539103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3539329Z 2022-11-23T02:22:33.3539422Z Running tests... 2022-11-23T02:22:33.3539839Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3541142Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:22:33.3541829Z 2022-11-23T02:22:33.3542113Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3542446Z Ran 1 test in 0.510s 2022-11-23T02:22:33.3542603Z 2022-11-23T02:22:33.3542696Z OK (skipped=1) 2022-11-23T02:22:33.3542845Z 2022-11-23T02:22:33.3542961Z Generating XML reports... 2022-11-23T02:22:33.3543611Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020226.xml 2022-11-23T02:22:33.3544311Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3544957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3545428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3546044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3546530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3546758Z 2022-11-23T02:22:33.3546859Z Running tests... 2022-11-23T02:22:33.3547295Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3547874Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34096 2022-11-23T02:22:33.3548467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34097 2022-11-23T02:22:33.3548986Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3549735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3550206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3550822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3551310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3551789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3552465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3552917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3553530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3554011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3554484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3555232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3555984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3556529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3557010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3557348Z ok (4.456s) 2022-11-23T02:22:33.3557491Z 2022-11-23T02:22:33.3557776Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3558104Z Ran 1 test in 4.457s 2022-11-23T02:22:33.3558265Z 2022-11-23T02:22:33.3558349Z OK 2022-11-23T02:22:33.3558474Z 2022-11-23T02:22:33.3558585Z Generating XML reports... 2022-11-23T02:22:33.3559247Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020229.xml 2022-11-23T02:22:33.3559963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3560674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3561144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3561764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3562247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3562475Z 2022-11-23T02:22:33.3562570Z Running tests... 2022-11-23T02:22:33.3562999Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3563503Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-11-23T02:22:33.3563800Z 2022-11-23T02:22:33.3564070Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3564404Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3564567Z 2022-11-23T02:22:33.3564664Z OK (skipped=1) 2022-11-23T02:22:33.3564808Z 2022-11-23T02:22:33.3564921Z Generating XML reports... 2022-11-23T02:22:33.3565576Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020237.xml 2022-11-23T02:22:33.3566285Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3566950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3567405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3568084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3568574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3568804Z 2022-11-23T02:22:33.3568905Z Running tests... 2022-11-23T02:22:33.3569341Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3569788Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3570343Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34340 2022-11-23T02:22:33.3570890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34341 2022-11-23T02:22:33.3571423Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3572123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3572599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3573235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3573788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3574265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3574959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3575412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3576216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3576707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3577185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3577896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3578653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3579205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3579687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3580208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwiylojbu 2022-11-23T02:22:33.3580786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwiylojbu/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3581345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30eorbq5 2022-11-23T02:22:33.3581908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30eorbq5/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3582304Z ok (6.848s) 2022-11-23T02:22:33.3582450Z 2022-11-23T02:22:33.3582741Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3583069Z Ran 1 test in 6.848s 2022-11-23T02:22:33.3583227Z 2022-11-23T02:22:33.3583312Z OK 2022-11-23T02:22:33.3583439Z 2022-11-23T02:22:33.3583556Z Generating XML reports... 2022-11-23T02:22:33.3584220Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020240.xml 2022-11-23T02:22:33.3584933Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3585600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3586072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3586761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3587256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3587490Z 2022-11-23T02:22:33.3587593Z Running tests... 2022-11-23T02:22:33.3588035Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3588511Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3589107Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34536 2022-11-23T02:22:33.3589676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34537 2022-11-23T02:22:33.3590212Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3590904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3591372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3592036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3592528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3593002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3593687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3594153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3594758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3595254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3595728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3596444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3597190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3597737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3598230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3598757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3ai0jpt 2022-11-23T02:22:33.3599308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3ai0jpt/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3599875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5w9t2egh 2022-11-23T02:22:33.3600495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5w9t2egh/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3601039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3601553Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3601918Z ok (6.948s) 2022-11-23T02:22:33.3602065Z 2022-11-23T02:22:33.3602359Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3602686Z Ran 1 test in 6.948s 2022-11-23T02:22:33.3602847Z 2022-11-23T02:22:33.3602932Z OK 2022-11-23T02:22:33.3603062Z 2022-11-23T02:22:33.3603180Z Generating XML reports... 2022-11-23T02:22:33.3603847Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020249.xml 2022-11-23T02:22:33.3604625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3605293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3605762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3606374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3606863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3607094Z 2022-11-23T02:22:33.3607197Z Running tests... 2022-11-23T02:22:33.3607634Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3608131Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3608758Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34732 2022-11-23T02:22:33.3609340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34733 2022-11-23T02:22:33.3609855Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3610618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3611090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3611712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3612203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3612677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3613355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3613832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3614442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3614931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3615398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3616110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3616859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3617401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3631535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3632073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9xvfdi2 2022-11-23T02:22:33.3632649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9xvfdi2/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3633215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7e3owr9 2022-11-23T02:22:33.3633785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7e3owr9/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3634323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3634828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.3635177Z ok (6.873s) 2022-11-23T02:22:33.3635321Z 2022-11-23T02:22:33.3635696Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3636039Z Ran 1 test in 6.873s 2022-11-23T02:22:33.3636348Z 2022-11-23T02:22:33.3636431Z OK 2022-11-23T02:22:33.3636557Z 2022-11-23T02:22:33.3636673Z Generating XML reports... 2022-11-23T02:22:33.3637345Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020300.xml 2022-11-23T02:22:33.3638053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3638721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3639196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3639820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3640372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3640601Z 2022-11-23T02:22:33.3640700Z Running tests... 2022-11-23T02:22:33.3641134Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3641593Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:22:33.3642181Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34928 2022-11-23T02:22:33.3642809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34929 2022-11-23T02:22:33.3643339Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3644048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3644507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3645120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3645609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3646077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3646750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3647211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3647822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3648299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3648764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3649471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3650215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3650755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3651238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3651761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcb_wtia 2022-11-23T02:22:33.3652326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcb_wtia/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3652876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4g9ay1c 2022-11-23T02:22:33.3653435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4g9ay1c/_remote_module_non_scriptable.py 2022-11-23T02:22:33.3653825Z ok (6.548s) 2022-11-23T02:22:33.3653969Z 2022-11-23T02:22:33.3654248Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3654640Z Ran 1 test in 6.548s 2022-11-23T02:22:33.3654800Z 2022-11-23T02:22:33.3654882Z OK 2022-11-23T02:22:33.3655009Z 2022-11-23T02:22:33.3655121Z Generating XML reports... 2022-11-23T02:22:33.3655784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020309.xml 2022-11-23T02:22:33.3656492Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3657162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3657613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3658229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3658724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3658949Z 2022-11-23T02:22:33.3659049Z Running tests... 2022-11-23T02:22:33.3659479Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3659974Z test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3660266Z 2022-11-23T02:22:33.3660549Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3660914Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3661073Z 2022-11-23T02:22:33.3661166Z OK (skipped=1) 2022-11-23T02:22:33.3661311Z 2022-11-23T02:22:33.3661425Z Generating XML reports... 2022-11-23T02:22:33.3662087Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020319.xml 2022-11-23T02:22:33.3662793Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3663462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3663929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3664538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3665023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3665248Z 2022-11-23T02:22:33.3665347Z Running tests... 2022-11-23T02:22:33.3665777Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3666312Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:22:33.3666642Z 2022-11-23T02:22:33.3666924Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3667254Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3667409Z 2022-11-23T02:22:33.3667504Z OK (skipped=1) 2022-11-23T02:22:33.3667641Z 2022-11-23T02:22:33.3667755Z Generating XML reports... 2022-11-23T02:22:33.3668406Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020322.xml 2022-11-23T02:22:33.3669114Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3669779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3670252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3670872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3671360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3671589Z 2022-11-23T02:22:33.3671675Z Running tests... 2022-11-23T02:22:33.3672101Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3672645Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:22:33.3673025Z 2022-11-23T02:22:33.3673310Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3673641Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3673798Z 2022-11-23T02:22:33.3673896Z OK (skipped=1) 2022-11-23T02:22:33.3674043Z 2022-11-23T02:22:33.3674157Z Generating XML reports... 2022-11-23T02:22:33.3674799Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020325.xml 2022-11-23T02:22:33.3675495Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3676305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3676773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3677390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3677884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3678113Z 2022-11-23T02:22:33.3678210Z Running tests... 2022-11-23T02:22:33.3678628Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3679207Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:22:33.3679532Z 2022-11-23T02:22:33.3679811Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3680204Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3680358Z 2022-11-23T02:22:33.3680455Z OK (skipped=1) 2022-11-23T02:22:33.3680605Z 2022-11-23T02:22:33.3680722Z Generating XML reports... 2022-11-23T02:22:33.3681379Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020328.xml 2022-11-23T02:22:33.3682074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3682743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3683205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3683828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3684317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3684550Z 2022-11-23T02:22:33.3684646Z Running tests... 2022-11-23T02:22:33.3685080Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3685606Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:22:33.3685933Z 2022-11-23T02:22:33.3686213Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3686546Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3686553Z 2022-11-23T02:22:33.3686648Z OK (skipped=1) 2022-11-23T02:22:33.3686654Z 2022-11-23T02:22:33.3686767Z Generating XML reports... 2022-11-23T02:22:33.3687259Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020330.xml 2022-11-23T02:22:33.3687595Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3687994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3688168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3688576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3688765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3688772Z 2022-11-23T02:22:33.3688870Z Running tests... 2022-11-23T02:22:33.3689215Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3689521Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s) 2022-11-23T02:22:33.3689528Z 2022-11-23T02:22:33.3689808Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3689906Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3689913Z 2022-11-23T02:22:33.3690007Z OK (skipped=1) 2022-11-23T02:22:33.3690013Z 2022-11-23T02:22:33.3690127Z Generating XML reports... 2022-11-23T02:22:33.3690614Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020333.xml 2022-11-23T02:22:33.3690945Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3691328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3691502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3691906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3692095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3692102Z 2022-11-23T02:22:33.3692249Z Running tests... 2022-11-23T02:22:33.3692534Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3692808Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3692815Z 2022-11-23T02:22:33.3693091Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3693194Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3693200Z 2022-11-23T02:22:33.3693298Z OK (skipped=1) 2022-11-23T02:22:33.3693304Z 2022-11-23T02:22:33.3693419Z Generating XML reports... 2022-11-23T02:22:33.3693910Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020336.xml 2022-11-23T02:22:33.3694245Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3694647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3694817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3695222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3695409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3695416Z 2022-11-23T02:22:33.3695513Z Running tests... 2022-11-23T02:22:33.3695795Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3696124Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35530 2022-11-23T02:22:33.3696345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35531 2022-11-23T02:22:33.3696623Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3697020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3697181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3697588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3697772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3698020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3698413Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3698635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3699044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3699233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3699479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3699911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3700339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3700573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3700801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3701179Z STAGE:2022-11-23 02:03:42 35530:35530 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3701535Z STAGE:2022-11-23 02:03:42 35531:35531 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3701941Z STAGE:2022-11-23 02:03:42 35530:35530 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3702298Z STAGE:2022-11-23 02:03:42 35531:35531 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3702676Z STAGE:2022-11-23 02:03:42 35530:35530 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3702917Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3703293Z STAGE:2022-11-23 02:03:42 35531:35531 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3703531Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3703894Z STAGE:2022-11-23 02:03:42 35530:35530 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3704252Z STAGE:2022-11-23 02:03:42 35530:35530 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3704617Z STAGE:2022-11-23 02:03:42 35530:35530 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3704972Z STAGE:2022-11-23 02:03:42 35531:35531 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3705332Z STAGE:2022-11-23 02:03:42 35531:35531 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3705705Z STAGE:2022-11-23 02:03:42 35531:35531 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3705797Z ok (4.242s) 2022-11-23T02:22:33.3705804Z 2022-11-23T02:22:33.3706089Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3706197Z Ran 1 test in 4.242s 2022-11-23T02:22:33.3706204Z 2022-11-23T02:22:33.3706286Z OK 2022-11-23T02:22:33.3706292Z 2022-11-23T02:22:33.3706406Z Generating XML reports... 2022-11-23T02:22:33.3706899Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020339.xml 2022-11-23T02:22:33.3707236Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3707631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3707804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3708213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3708402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3708409Z 2022-11-23T02:22:33.3708553Z Running tests... 2022-11-23T02:22:33.3708839Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3709178Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35722 2022-11-23T02:22:33.3709403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35723 2022-11-23T02:22:33.3709681Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3710073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3710247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3710642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3710829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3711083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3711478Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3711713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3712121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3712306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3712552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3712981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3713407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3713650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3713874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3714238Z STAGE:2022-11-23 02:03:49 35722:35722 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3714589Z STAGE:2022-11-23 02:03:49 35723:35723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3714950Z STAGE:2022-11-23 02:03:49 35722:35722 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3715322Z STAGE:2022-11-23 02:03:49 35722:35722 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3715562Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3715916Z STAGE:2022-11-23 02:03:49 35723:35723 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3716286Z STAGE:2022-11-23 02:03:49 35723:35723 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3716522Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3716881Z STAGE:2022-11-23 02:03:49 35722:35722 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3717237Z STAGE:2022-11-23 02:03:49 35722:35722 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3717603Z STAGE:2022-11-23 02:03:49 35722:35722 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3717958Z STAGE:2022-11-23 02:03:49 35723:35723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3718308Z STAGE:2022-11-23 02:03:49 35723:35723 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3718680Z STAGE:2022-11-23 02:03:49 35723:35723 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3718820Z ok (4.243s) 2022-11-23T02:22:33.3718827Z 2022-11-23T02:22:33.3719114Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3719217Z Ran 1 test in 4.244s 2022-11-23T02:22:33.3719224Z 2022-11-23T02:22:33.3719307Z OK 2022-11-23T02:22:33.3719317Z 2022-11-23T02:22:33.3719431Z Generating XML reports... 2022-11-23T02:22:33.3719920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020346.xml 2022-11-23T02:22:33.3720322Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3720723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3720895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3721301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3721492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3721499Z 2022-11-23T02:22:33.3721596Z Running tests... 2022-11-23T02:22:33.3721879Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3722210Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3722217Z 2022-11-23T02:22:33.3722495Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3722598Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3722605Z 2022-11-23T02:22:33.3722701Z OK (skipped=1) 2022-11-23T02:22:33.3722707Z 2022-11-23T02:22:33.3722823Z Generating XML reports... 2022-11-23T02:22:33.3723308Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020354.xml 2022-11-23T02:22:33.3723646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3724029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3724198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3724611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3724800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3724807Z 2022-11-23T02:22:33.3724905Z Running tests... 2022-11-23T02:22:33.3725184Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3725454Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3725461Z 2022-11-23T02:22:33.3725743Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3725846Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3725853Z 2022-11-23T02:22:33.3725949Z OK (skipped=1) 2022-11-23T02:22:33.3725955Z 2022-11-23T02:22:33.3726072Z Generating XML reports... 2022-11-23T02:22:33.3726562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020357.xml 2022-11-23T02:22:33.3726900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3727297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3727467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3727873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3728059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3728066Z 2022-11-23T02:22:33.3728214Z Running tests... 2022-11-23T02:22:33.3728497Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3728843Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36030 2022-11-23T02:22:33.3729063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36031 2022-11-23T02:22:33.3729337Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3729731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3729891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3730294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3730484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3730733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3731123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3731336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3731748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3731934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3732183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3732615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3733039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3733267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3733483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3733853Z STAGE:2022-11-23 02:04:02 36031:36031 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3734204Z STAGE:2022-11-23 02:04:02 36030:36030 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3734566Z STAGE:2022-11-23 02:04:02 36031:36031 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3734942Z STAGE:2022-11-23 02:04:02 36031:36031 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3735184Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3735542Z STAGE:2022-11-23 02:04:02 36030:36030 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3735917Z STAGE:2022-11-23 02:04:02 36030:36030 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3736154Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3736241Z ok (4.354s) 2022-11-23T02:22:33.3736250Z 2022-11-23T02:22:33.3736532Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3736623Z Ran 1 test in 4.355s 2022-11-23T02:22:33.3736629Z 2022-11-23T02:22:33.3736706Z OK 2022-11-23T02:22:33.3736713Z 2022-11-23T02:22:33.3736825Z Generating XML reports... 2022-11-23T02:22:33.3737315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020400.xml 2022-11-23T02:22:33.3737652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3738051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3738275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3738683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3738871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3738878Z 2022-11-23T02:22:33.3738971Z Running tests... 2022-11-23T02:22:33.3739251Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3739597Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36222 2022-11-23T02:22:33.3739811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36223 2022-11-23T02:22:33.3740085Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3740482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3740651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3741099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3741289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3741540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3741938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3742108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3742515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3742693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3742936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3743370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3743802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3744035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3744253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3744616Z STAGE:2022-11-23 02:04:09 36223:36223 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3744966Z STAGE:2022-11-23 02:04:10 36222:36222 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3745328Z STAGE:2022-11-23 02:04:10 36223:36223 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3745676Z STAGE:2022-11-23 02:04:10 36222:36222 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3746053Z STAGE:2022-11-23 02:04:10 36223:36223 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3746669Z STAGE:2022-11-23 02:04:10 36222:36222 ActivityProfilerController.cpp:310] Completed Stage: Post Processing[W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3746677Z 2022-11-23T02:22:33.3746915Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3747004Z ok (4.240s) 2022-11-23T02:22:33.3747011Z 2022-11-23T02:22:33.3747296Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3747399Z Ran 1 test in 4.241s 2022-11-23T02:22:33.3747451Z 2022-11-23T02:22:33.3747528Z OK 2022-11-23T02:22:33.3747535Z 2022-11-23T02:22:33.3747648Z Generating XML reports... 2022-11-23T02:22:33.3748142Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020407.xml 2022-11-23T02:22:33.3748475Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3748871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3749043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3749447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3749630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3749636Z 2022-11-23T02:22:33.3749730Z Running tests... 2022-11-23T02:22:33.3750007Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3751047Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:22:33.3751070Z 2022-11-23T02:22:33.3751354Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3751445Z Ran 1 test in 0.510s 2022-11-23T02:22:33.3751452Z 2022-11-23T02:22:33.3751549Z OK (skipped=1) 2022-11-23T02:22:33.3751555Z 2022-11-23T02:22:33.3751665Z Generating XML reports... 2022-11-23T02:22:33.3752150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020414.xml 2022-11-23T02:22:33.3752479Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3752874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3753044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3753456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3753640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3753647Z 2022-11-23T02:22:33.3753741Z Running tests... 2022-11-23T02:22:33.3754021Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3755029Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.523s) 2022-11-23T02:22:33.3755040Z 2022-11-23T02:22:33.3755314Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3755415Z Ran 1 test in 0.523s 2022-11-23T02:22:33.3755422Z 2022-11-23T02:22:33.3755520Z OK (skipped=1) 2022-11-23T02:22:33.3755529Z 2022-11-23T02:22:33.3755641Z Generating XML reports... 2022-11-23T02:22:33.3756132Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020418.xml 2022-11-23T02:22:33.3756467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3756865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3757038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3757445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3757674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3757681Z 2022-11-23T02:22:33.3757776Z Running tests... 2022-11-23T02:22:33.3758061Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3758406Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36530 2022-11-23T02:22:33.3758615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36531 2022-11-23T02:22:33.3758888Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3759279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3759446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3759855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3760042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3760404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3760801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3760972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3761376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3761565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3761815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3762245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3762670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3762905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3763127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3763217Z ok (4.353s) 2022-11-23T02:22:33.3763225Z 2022-11-23T02:22:33.3763507Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3763610Z Ran 1 test in 4.353s 2022-11-23T02:22:33.3763617Z 2022-11-23T02:22:33.3763697Z OK 2022-11-23T02:22:33.3763704Z 2022-11-23T02:22:33.3763818Z Generating XML reports... 2022-11-23T02:22:33.3764308Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020421.xml 2022-11-23T02:22:33.3764634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3765026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3765197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3765597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3765783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3765790Z 2022-11-23T02:22:33.3765886Z Running tests... 2022-11-23T02:22:33.3766164Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3766505Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36716 2022-11-23T02:22:33.3766777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36717 2022-11-23T02:22:33.3767055Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3767455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3767628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3768034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3768219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3768464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3768852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3769025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3769434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3769621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3769907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3770340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3770763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3770992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3771210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3771449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.3771691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.3772123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.3772544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.3772785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:22:33.3773206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:22:33.3773440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:22:33.3773857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:22:33.3774099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T02:22:33.3774333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T02:22:33.3774908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:22:33.3775320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:22:33.3775412Z ok (4.944s) 2022-11-23T02:22:33.3775419Z 2022-11-23T02:22:33.3775698Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3775798Z Ran 1 test in 4.944s 2022-11-23T02:22:33.3775804Z 2022-11-23T02:22:33.3775886Z OK 2022-11-23T02:22:33.3775892Z 2022-11-23T02:22:33.3776006Z Generating XML reports... 2022-11-23T02:22:33.3776549Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020429.xml 2022-11-23T02:22:33.3776884Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3777283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3777456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3777863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3778051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3778058Z 2022-11-23T02:22:33.3778153Z Running tests... 2022-11-23T02:22:33.3778423Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3778748Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36937 2022-11-23T02:22:33.3778973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36938 2022-11-23T02:22:33.3779250Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3779686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3779856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3780265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3780454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3780703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3781100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3781275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3781682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3781870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3782116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3782546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3782974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3783205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3783431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3783524Z ok (4.543s) 2022-11-23T02:22:33.3783531Z 2022-11-23T02:22:33.3783816Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3783915Z Ran 1 test in 4.544s 2022-11-23T02:22:33.3783921Z 2022-11-23T02:22:33.3784005Z OK 2022-11-23T02:22:33.3784015Z 2022-11-23T02:22:33.3784118Z Generating XML reports... 2022-11-23T02:22:33.3784615Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020437.xml 2022-11-23T02:22:33.3784950Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3785342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3785517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3785923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3786215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3786221Z 2022-11-23T02:22:33.3786318Z Running tests... 2022-11-23T02:22:33.3786604Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3787073Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:22:33.3787080Z 2022-11-23T02:22:33.3787362Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3787466Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3787472Z 2022-11-23T02:22:33.3787570Z OK (skipped=1) 2022-11-23T02:22:33.3787576Z 2022-11-23T02:22:33.3787687Z Generating XML reports... 2022-11-23T02:22:33.3788173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020444.xml 2022-11-23T02:22:33.3788509Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3788910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3789084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3789535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3789729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3789735Z 2022-11-23T02:22:33.3789830Z Running tests... 2022-11-23T02:22:33.3790113Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3790560Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:22:33.3790578Z 2022-11-23T02:22:33.3790844Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3790949Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3790956Z 2022-11-23T02:22:33.3791051Z OK (skipped=1) 2022-11-23T02:22:33.3791057Z 2022-11-23T02:22:33.3791172Z Generating XML reports... 2022-11-23T02:22:33.3791663Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020447.xml 2022-11-23T02:22:33.3792000Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3792398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3792570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3792974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3793165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3793174Z 2022-11-23T02:22:33.3793274Z Running tests... 2022-11-23T02:22:33.3793555Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3794031Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:22:33.3794041Z 2022-11-23T02:22:33.3794309Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3794409Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3794415Z 2022-11-23T02:22:33.3794508Z OK (skipped=1) 2022-11-23T02:22:33.3794515Z 2022-11-23T02:22:33.3794626Z Generating XML reports... 2022-11-23T02:22:33.3795115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020450.xml 2022-11-23T02:22:33.3795448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3795841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3796057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3796466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3796659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3796665Z 2022-11-23T02:22:33.3796753Z Running tests... 2022-11-23T02:22:33.3797035Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3797492Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:22:33.3797499Z 2022-11-23T02:22:33.3797775Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3797874Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3797880Z 2022-11-23T02:22:33.3797975Z OK (skipped=1) 2022-11-23T02:22:33.3797981Z 2022-11-23T02:22:33.3798094Z Generating XML reports... 2022-11-23T02:22:33.3798577Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020453.xml 2022-11-23T02:22:33.3798914Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3799354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3799528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3799938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3800197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3800204Z 2022-11-23T02:22:33.3800298Z Running tests... 2022-11-23T02:22:33.3800582Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3801038Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:22:33.3801046Z 2022-11-23T02:22:33.3801322Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3801422Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3801431Z 2022-11-23T02:22:33.3801526Z OK (skipped=1) 2022-11-23T02:22:33.3801532Z 2022-11-23T02:22:33.3801646Z Generating XML reports... 2022-11-23T02:22:33.3802131Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020456.xml 2022-11-23T02:22:33.3802466Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3802860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3803020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3803428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3803613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3803620Z 2022-11-23T02:22:33.3803715Z Running tests... 2022-11-23T02:22:33.3803999Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3804446Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:22:33.3804452Z 2022-11-23T02:22:33.3804734Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3804834Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3804840Z 2022-11-23T02:22:33.3804936Z OK (skipped=1) 2022-11-23T02:22:33.3804942Z 2022-11-23T02:22:33.3805054Z Generating XML reports... 2022-11-23T02:22:33.3805537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020459.xml 2022-11-23T02:22:33.3805939Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3806332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3806503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3806905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3807090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3807096Z 2022-11-23T02:22:33.3807193Z Running tests... 2022-11-23T02:22:33.3807471Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3807930Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:22:33.3807941Z 2022-11-23T02:22:33.3808223Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3808326Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3808332Z 2022-11-23T02:22:33.3808428Z OK (skipped=1) 2022-11-23T02:22:33.3808434Z 2022-11-23T02:22:33.3808548Z Generating XML reports... 2022-11-23T02:22:33.3809071Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020501.xml 2022-11-23T02:22:33.3809412Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3809811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3809980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3810392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3810582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3810588Z 2022-11-23T02:22:33.3810684Z Running tests... 2022-11-23T02:22:33.3810967Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3811424Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:22:33.3811431Z 2022-11-23T02:22:33.3811707Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3811809Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3811815Z 2022-11-23T02:22:33.3811912Z OK (skipped=1) 2022-11-23T02:22:33.3811918Z 2022-11-23T02:22:33.3812032Z Generating XML reports... 2022-11-23T02:22:33.3812520Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020504.xml 2022-11-23T02:22:33.3812856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3813259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3813431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3813843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3814033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3814039Z 2022-11-23T02:22:33.3814136Z Running tests... 2022-11-23T02:22:33.3814414Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3814848Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:22:33.3814855Z 2022-11-23T02:22:33.3815134Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3815223Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3815282Z 2022-11-23T02:22:33.3815367Z OK (skipped=1) 2022-11-23T02:22:33.3815374Z 2022-11-23T02:22:33.3815487Z Generating XML reports... 2022-11-23T02:22:33.3815974Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020507.xml 2022-11-23T02:22:33.3816312Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3816710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3816879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3817280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3817466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3817473Z 2022-11-23T02:22:33.3817569Z Running tests... 2022-11-23T02:22:33.3817856Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3818174Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3818182Z 2022-11-23T02:22:33.3818458Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3818602Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3818610Z 2022-11-23T02:22:33.3818707Z OK (skipped=1) 2022-11-23T02:22:33.3818713Z 2022-11-23T02:22:33.3818824Z Generating XML reports... 2022-11-23T02:22:33.3819311Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020510.xml 2022-11-23T02:22:33.3819646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3820043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3820217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3820624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3820811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3820818Z 2022-11-23T02:22:33.3820919Z Running tests... 2022-11-23T02:22:33.3821187Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3821623Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:22:33.3821630Z 2022-11-23T02:22:33.3821907Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3822008Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3822014Z 2022-11-23T02:22:33.3822108Z OK (skipped=1) 2022-11-23T02:22:33.3822114Z 2022-11-23T02:22:33.3822232Z Generating XML reports... 2022-11-23T02:22:33.3822718Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020513.xml 2022-11-23T02:22:33.3823055Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3823452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3823623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3824035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3824223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3824229Z 2022-11-23T02:22:33.3824326Z Running tests... 2022-11-23T02:22:33.3824604Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3825050Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:22:33.3825106Z 2022-11-23T02:22:33.3825388Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3825488Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3825495Z 2022-11-23T02:22:33.3825591Z OK (skipped=1) 2022-11-23T02:22:33.3825597Z 2022-11-23T02:22:33.3825711Z Generating XML reports... 2022-11-23T02:22:33.3826200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020516.xml 2022-11-23T02:22:33.3826532Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3826925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3827097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3827498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3827685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3827693Z 2022-11-23T02:22:33.3827791Z Running tests... 2022-11-23T02:22:33.3828059Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3828614Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:22:33.3828621Z 2022-11-23T02:22:33.3828902Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3829002Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3829009Z 2022-11-23T02:22:33.3829103Z OK (skipped=1) 2022-11-23T02:22:33.3829109Z 2022-11-23T02:22:33.3829221Z Generating XML reports... 2022-11-23T02:22:33.3829706Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020519.xml 2022-11-23T02:22:33.3830042Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3830438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3830605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3831015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3831202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3831209Z 2022-11-23T02:22:33.3831307Z Running tests... 2022-11-23T02:22:33.3831589Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3831888Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.3831895Z 2022-11-23T02:22:33.3832170Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3832271Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3832278Z 2022-11-23T02:22:33.3832377Z OK (skipped=1) 2022-11-23T02:22:33.3832383Z 2022-11-23T02:22:33.3832502Z Generating XML reports... 2022-11-23T02:22:33.3832983Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020522.xml 2022-11-23T02:22:33.3833317Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3833713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3833884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3834276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3834465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3834471Z 2022-11-23T02:22:33.3834566Z Running tests... 2022-11-23T02:22:33.3834892Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3835170Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3835177Z 2022-11-23T02:22:33.3835461Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3835564Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3835571Z 2022-11-23T02:22:33.3835672Z OK (skipped=1) 2022-11-23T02:22:33.3835678Z 2022-11-23T02:22:33.3835791Z Generating XML reports... 2022-11-23T02:22:33.3836280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020525.xml 2022-11-23T02:22:33.3836617Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3837015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3837191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3837599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3837789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3837795Z 2022-11-23T02:22:33.3837936Z Running tests... 2022-11-23T02:22:33.3838222Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3838502Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3838508Z 2022-11-23T02:22:33.3838786Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3838889Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3838896Z 2022-11-23T02:22:33.3838992Z OK (skipped=1) 2022-11-23T02:22:33.3838998Z 2022-11-23T02:22:33.3839113Z Generating XML reports... 2022-11-23T02:22:33.3839605Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020528.xml 2022-11-23T02:22:33.3839926Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3840385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3840559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3840969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3841151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3841158Z 2022-11-23T02:22:33.3841252Z Running tests... 2022-11-23T02:22:33.3841530Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3841815Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.3841824Z 2022-11-23T02:22:33.3842102Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3842202Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3842208Z 2022-11-23T02:22:33.3842303Z OK (skipped=1) 2022-11-23T02:22:33.3842309Z 2022-11-23T02:22:33.3842427Z Generating XML reports... 2022-11-23T02:22:33.3842916Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020530.xml 2022-11-23T02:22:33.3843252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3843652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3843826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3844235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3844483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3844489Z 2022-11-23T02:22:33.3844586Z Running tests... 2022-11-23T02:22:33.3844869Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3845151Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.3845159Z 2022-11-23T02:22:33.3845436Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3845526Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3845544Z 2022-11-23T02:22:33.3845629Z OK (skipped=1) 2022-11-23T02:22:33.3845645Z 2022-11-23T02:22:33.3845747Z Generating XML reports... 2022-11-23T02:22:33.3846227Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020533.xml 2022-11-23T02:22:33.3846555Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3846950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3847120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3847576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3847763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3847770Z 2022-11-23T02:22:33.3847868Z Running tests... 2022-11-23T02:22:33.3848149Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3848424Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3848430Z 2022-11-23T02:22:33.3848706Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3848807Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3848816Z 2022-11-23T02:22:33.3848913Z OK (skipped=1) 2022-11-23T02:22:33.3848919Z 2022-11-23T02:22:33.3849033Z Generating XML reports... 2022-11-23T02:22:33.3849521Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020536.xml 2022-11-23T02:22:33.3849861Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3850258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3850431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3850840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3851028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3851034Z 2022-11-23T02:22:33.3851130Z Running tests... 2022-11-23T02:22:33.3851403Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3851677Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3851684Z 2022-11-23T02:22:33.3851958Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3852060Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3852067Z 2022-11-23T02:22:33.3852161Z OK (skipped=1) 2022-11-23T02:22:33.3852167Z 2022-11-23T02:22:33.3852283Z Generating XML reports... 2022-11-23T02:22:33.3852767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020539.xml 2022-11-23T02:22:33.3853100Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3853497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3853670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3854122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3854306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3854312Z 2022-11-23T02:22:33.3854409Z Running tests... 2022-11-23T02:22:33.3854692Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3854972Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.3854978Z 2022-11-23T02:22:33.3855287Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3855389Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3855395Z 2022-11-23T02:22:33.3855487Z OK (skipped=1) 2022-11-23T02:22:33.3855493Z 2022-11-23T02:22:33.3855606Z Generating XML reports... 2022-11-23T02:22:33.3856087Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020542.xml 2022-11-23T02:22:33.3856425Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3856817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3857156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3857549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3857737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3857743Z 2022-11-23T02:22:33.3857842Z Running tests... 2022-11-23T02:22:33.3858124Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3858396Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.3858405Z 2022-11-23T02:22:33.3858682Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3858787Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3858793Z 2022-11-23T02:22:33.3858892Z OK (skipped=1) 2022-11-23T02:22:33.3858899Z 2022-11-23T02:22:33.3859011Z Generating XML reports... 2022-11-23T02:22:33.3859502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020545.xml 2022-11-23T02:22:33.3859838Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3860228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3860396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3860803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3860997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3861003Z 2022-11-23T02:22:33.3861102Z Running tests... 2022-11-23T02:22:33.3861386Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3861658Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3861665Z 2022-11-23T02:22:33.3861943Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3862048Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3862054Z 2022-11-23T02:22:33.3862153Z OK (skipped=1) 2022-11-23T02:22:33.3862159Z 2022-11-23T02:22:33.3862274Z Generating XML reports... 2022-11-23T02:22:33.3862749Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020548.xml 2022-11-23T02:22:33.3863081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3863531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3863703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3864109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3864300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3864306Z 2022-11-23T02:22:33.3864400Z Running tests... 2022-11-23T02:22:33.3864683Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3864952Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.3864958Z 2022-11-23T02:22:33.3865237Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3865340Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3865346Z 2022-11-23T02:22:33.3865447Z OK (skipped=1) 2022-11-23T02:22:33.3865453Z 2022-11-23T02:22:33.3865566Z Generating XML reports... 2022-11-23T02:22:33.3866051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020551.xml 2022-11-23T02:22:33.3866430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3866829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3866998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3867405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3867594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3867600Z 2022-11-23T02:22:33.3867697Z Running tests... 2022-11-23T02:22:33.3867976Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3868270Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s) 2022-11-23T02:22:33.3868277Z 2022-11-23T02:22:33.3868553Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3868642Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3868664Z 2022-11-23T02:22:33.3868749Z OK (skipped=1) 2022-11-23T02:22:33.3868755Z 2022-11-23T02:22:33.3868872Z Generating XML reports... 2022-11-23T02:22:33.3869356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020554.xml 2022-11-23T02:22:33.3869691Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3870090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3870259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3870666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3870852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3870859Z 2022-11-23T02:22:33.3870955Z Running tests... 2022-11-23T02:22:33.3871239Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3871537Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s) 2022-11-23T02:22:33.3871544Z 2022-11-23T02:22:33.3871816Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3871917Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3871923Z 2022-11-23T02:22:33.3872014Z OK (skipped=1) 2022-11-23T02:22:33.3872020Z 2022-11-23T02:22:33.3872134Z Generating XML reports... 2022-11-23T02:22:33.3872614Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020557.xml 2022-11-23T02:22:33.3872996Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3873394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3873570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3873977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3874164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3874170Z 2022-11-23T02:22:33.3874268Z Running tests... 2022-11-23T02:22:33.3874539Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3874821Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.3874827Z 2022-11-23T02:22:33.3875100Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3875202Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3875208Z 2022-11-23T02:22:33.3875304Z OK (skipped=1) 2022-11-23T02:22:33.3875310Z 2022-11-23T02:22:33.3875427Z Generating XML reports... 2022-11-23T02:22:33.3875955Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020600.xml 2022-11-23T02:22:33.3876296Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3876690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3876861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3877265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3877452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3877461Z 2022-11-23T02:22:33.3877557Z Running tests... 2022-11-23T02:22:33.3877836Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3878174Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38693 2022-11-23T02:22:33.3878390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38694 2022-11-23T02:22:33.3878667Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3879057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3879224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3879631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3879815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3880065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3880528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3880688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3881093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3881282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3881530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3881964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3882457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3882689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3882919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3883013Z ok (4.245s) 2022-11-23T02:22:33.3883019Z 2022-11-23T02:22:33.3883302Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3883401Z Ran 1 test in 4.246s 2022-11-23T02:22:33.3883408Z 2022-11-23T02:22:33.3883488Z OK 2022-11-23T02:22:33.3883494Z 2022-11-23T02:22:33.3883607Z Generating XML reports... 2022-11-23T02:22:33.3884103Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020602.xml 2022-11-23T02:22:33.3884434Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3884834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3885006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3885459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3885653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3885660Z 2022-11-23T02:22:33.3885761Z Running tests... 2022-11-23T02:22:33.3886044Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3886301Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.3886319Z 2022-11-23T02:22:33.3886587Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3886690Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3886696Z 2022-11-23T02:22:33.3886795Z OK (skipped=1) 2022-11-23T02:22:33.3886802Z 2022-11-23T02:22:33.3886916Z Generating XML reports... 2022-11-23T02:22:33.3887404Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020610.xml 2022-11-23T02:22:33.3887743Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3888135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3888309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3888715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3888904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3888911Z 2022-11-23T02:22:33.3889009Z Running tests... 2022-11-23T02:22:33.3889290Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3889570Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.3889578Z 2022-11-23T02:22:33.3889853Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3889959Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3889969Z 2022-11-23T02:22:33.3890067Z OK (skipped=1) 2022-11-23T02:22:33.3890073Z 2022-11-23T02:22:33.3890186Z Generating XML reports... 2022-11-23T02:22:33.3890672Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020613.xml 2022-11-23T02:22:33.3891008Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3891398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3891571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3892024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3892198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3892220Z 2022-11-23T02:22:33.3892306Z Running tests... 2022-11-23T02:22:33.3892595Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3892874Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.3892881Z 2022-11-23T02:22:33.3893157Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3893257Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3893264Z 2022-11-23T02:22:33.3893361Z OK (skipped=1) 2022-11-23T02:22:33.3893367Z 2022-11-23T02:22:33.3893477Z Generating XML reports... 2022-11-23T02:22:33.3893961Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020615.xml 2022-11-23T02:22:33.3894294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3894684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3894899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3895309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3895492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3895499Z 2022-11-23T02:22:33.3895597Z Running tests... 2022-11-23T02:22:33.3895879Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3896214Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39053 2022-11-23T02:22:33.3896436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39054 2022-11-23T02:22:33.3896710Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3897105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3897278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3897694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3897883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3898120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3898513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3898683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3899091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3899283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3899540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3899974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3900405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3900640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3900872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3901291Z STAGE:2022-11-23 02:06:21 39054:39054 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3901643Z STAGE:2022-11-23 02:06:21 39053:39053 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3902011Z STAGE:2022-11-23 02:06:21 39053:39053 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3902390Z STAGE:2022-11-23 02:06:21 39053:39053 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3902630Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3902989Z STAGE:2022-11-23 02:06:21 39054:39054 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3903362Z STAGE:2022-11-23 02:06:21 39054:39054 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3903592Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3903947Z STAGE:2022-11-23 02:06:21 39054:39054 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3904292Z STAGE:2022-11-23 02:06:21 39053:39053 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3904699Z STAGE:2022-11-23 02:06:21 39054:39054 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3905083Z STAGE:2022-11-23 02:06:21 39054:39054 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3905449Z STAGE:2022-11-23 02:06:21 39053:39053 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3905826Z STAGE:2022-11-23 02:06:21 39053:39053 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3906171Z STAGE:2022-11-23 02:06:21 39054:39054 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3906533Z STAGE:2022-11-23 02:06:21 39054:39054 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3906910Z STAGE:2022-11-23 02:06:21 39054:39054 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3907265Z STAGE:2022-11-23 02:06:21 39053:39053 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3907631Z STAGE:2022-11-23 02:06:21 39053:39053 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3908004Z STAGE:2022-11-23 02:06:21 39053:39053 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3908096Z ok (4.553s) 2022-11-23T02:22:33.3908103Z 2022-11-23T02:22:33.3908386Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3908489Z Ran 1 test in 4.554s 2022-11-23T02:22:33.3908496Z 2022-11-23T02:22:33.3908581Z OK 2022-11-23T02:22:33.3908587Z 2022-11-23T02:22:33.3908704Z Generating XML reports... 2022-11-23T02:22:33.3909196Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020618.xml 2022-11-23T02:22:33.3909541Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3909940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3910118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3910532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3910727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3910733Z 2022-11-23T02:22:33.3910836Z Running tests... 2022-11-23T02:22:33.3911120Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3911467Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39245 2022-11-23T02:22:33.3911737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39246 2022-11-23T02:22:33.3912012Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3912407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3912570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3912976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3913167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3913415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3913808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3913983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3914391Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3914577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3914871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3915305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3915737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3915973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3916206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3916575Z STAGE:2022-11-23 02:06:29 39245:39245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3916932Z STAGE:2022-11-23 02:06:29 39246:39246 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3917301Z STAGE:2022-11-23 02:06:29 39246:39246 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3917679Z STAGE:2022-11-23 02:06:29 39246:39246 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3917924Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3918289Z STAGE:2022-11-23 02:06:29 39245:39245 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3918663Z STAGE:2022-11-23 02:06:29 39245:39245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3918900Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3919266Z STAGE:2022-11-23 02:06:29 39246:39246 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3919614Z STAGE:2022-11-23 02:06:29 39245:39245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3919963Z STAGE:2022-11-23 02:06:29 39246:39246 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3920404Z STAGE:2022-11-23 02:06:29 39246:39246 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3920769Z STAGE:2022-11-23 02:06:29 39245:39245 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3921143Z STAGE:2022-11-23 02:06:29 39245:39245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3921495Z STAGE:2022-11-23 02:06:29 39245:39245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3921836Z STAGE:2022-11-23 02:06:29 39246:39246 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3922264Z STAGE:2022-11-23 02:06:29 39245:39245 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3922643Z STAGE:2022-11-23 02:06:29 39245:39245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3923002Z STAGE:2022-11-23 02:06:29 39246:39246 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3923374Z STAGE:2022-11-23 02:06:29 39246:39246 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3923469Z ok (4.354s) 2022-11-23T02:22:33.3923476Z 2022-11-23T02:22:33.3923763Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3923867Z Ran 1 test in 4.355s 2022-11-23T02:22:33.3923875Z 2022-11-23T02:22:33.3923957Z OK 2022-11-23T02:22:33.3923963Z 2022-11-23T02:22:33.3924078Z Generating XML reports... 2022-11-23T02:22:33.3924576Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020626.xml 2022-11-23T02:22:33.3924913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3925491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3925721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3926137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3926325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3926331Z 2022-11-23T02:22:33.3926432Z Running tests... 2022-11-23T02:22:33.3926702Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3927051Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39437 2022-11-23T02:22:33.3927283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39438 2022-11-23T02:22:33.3927558Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3927955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3928128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3928536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3928728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3928978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3929373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3929544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3929948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3930138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3930392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3930826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3931254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3931488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3931716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3932157Z STAGE:2022-11-23 02:06:36 39437:39437 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3932507Z STAGE:2022-11-23 02:06:36 39438:39438 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3932876Z STAGE:2022-11-23 02:06:36 39438:39438 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3933253Z STAGE:2022-11-23 02:06:36 39438:39438 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3933501Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3933856Z STAGE:2022-11-23 02:06:36 39437:39437 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3934216Z STAGE:2022-11-23 02:06:36 39437:39437 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3934455Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.3934814Z STAGE:2022-11-23 02:06:36 39438:39438 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3935163Z STAGE:2022-11-23 02:06:36 39437:39437 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3935566Z STAGE:2022-11-23 02:06:36 39438:39438 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3935947Z STAGE:2022-11-23 02:06:36 39438:39438 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3936313Z STAGE:2022-11-23 02:06:36 39437:39437 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3936689Z STAGE:2022-11-23 02:06:36 39437:39437 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3937049Z STAGE:2022-11-23 02:06:36 39438:39438 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3937401Z STAGE:2022-11-23 02:06:36 39437:39437 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.3937769Z STAGE:2022-11-23 02:06:36 39438:39438 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3938146Z STAGE:2022-11-23 02:06:36 39438:39438 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3938514Z STAGE:2022-11-23 02:06:36 39437:39437 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.3938884Z STAGE:2022-11-23 02:06:36 39437:39437 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.3938978Z ok (4.253s) 2022-11-23T02:22:33.3938986Z 2022-11-23T02:22:33.3939272Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3939384Z Ran 1 test in 4.254s 2022-11-23T02:22:33.3939390Z 2022-11-23T02:22:33.3939468Z OK 2022-11-23T02:22:33.3939473Z 2022-11-23T02:22:33.3939588Z Generating XML reports... 2022-11-23T02:22:33.3940080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020633.xml 2022-11-23T02:22:33.3940421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3940821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3940984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3941392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3941584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3941590Z 2022-11-23T02:22:33.3941690Z Running tests... 2022-11-23T02:22:33.3941972Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3942220Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:22:33.3942227Z 2022-11-23T02:22:33.3942566Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3942669Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3942675Z 2022-11-23T02:22:33.3942774Z OK (skipped=1) 2022-11-23T02:22:33.3942779Z 2022-11-23T02:22:33.3942895Z Generating XML reports... 2022-11-23T02:22:33.3943391Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020641.xml 2022-11-23T02:22:33.3943729Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3944124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3944296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3944706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3944899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3944909Z 2022-11-23T02:22:33.3945010Z Running tests... 2022-11-23T02:22:33.3945296Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3945562Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:22:33.3945610Z 2022-11-23T02:22:33.3945892Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3945995Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3946002Z 2022-11-23T02:22:33.3946099Z OK (skipped=1) 2022-11-23T02:22:33.3946105Z 2022-11-23T02:22:33.3946219Z Generating XML reports... 2022-11-23T02:22:33.3946692Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020644.xml 2022-11-23T02:22:33.3947028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3947427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3947603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3948013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3948206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3948212Z 2022-11-23T02:22:33.3948314Z Running tests... 2022-11-23T02:22:33.3948597Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3948922Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39745 2022-11-23T02:22:33.3949140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39746 2022-11-23T02:22:33.3949417Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3949818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3949991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3950405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3950594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3950845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3951248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3951424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3951828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3952064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3952314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3952755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3953187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3953407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3953640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3953774Z skip: Test skipped for ROCm (3.741s) 2022-11-23T02:22:33.3953780Z 2022-11-23T02:22:33.3954067Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3954169Z Ran 1 test in 3.742s 2022-11-23T02:22:33.3954179Z 2022-11-23T02:22:33.3954276Z OK (skipped=1) 2022-11-23T02:22:33.3954282Z 2022-11-23T02:22:33.3954397Z Generating XML reports... 2022-11-23T02:22:33.3954886Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020647.xml 2022-11-23T02:22:33.3955265Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3955703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3955876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3956287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3956479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3956486Z 2022-11-23T02:22:33.3956587Z Running tests... 2022-11-23T02:22:33.3956875Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3957215Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39924 2022-11-23T02:22:33.3957439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39925 2022-11-23T02:22:33.3957717Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3958109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3958288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3958699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3958887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3959125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3959514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3959689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3960099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3960364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3960621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3961054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3961486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3961783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3962009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3962146Z skip: Test skipped for ROCm (3.938s) 2022-11-23T02:22:33.3962152Z 2022-11-23T02:22:33.3962445Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3962549Z Ran 1 test in 3.939s 2022-11-23T02:22:33.3962556Z 2022-11-23T02:22:33.3962652Z OK (skipped=1) 2022-11-23T02:22:33.3962659Z 2022-11-23T02:22:33.3962776Z Generating XML reports... 2022-11-23T02:22:33.3963272Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020654.xml 2022-11-23T02:22:33.3963614Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3964016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3964195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3964605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3964845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3964853Z 2022-11-23T02:22:33.3964953Z Running tests... 2022-11-23T02:22:33.3965228Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3965495Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:22:33.3965515Z 2022-11-23T02:22:33.3965784Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3965889Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3965895Z 2022-11-23T02:22:33.3965997Z OK (skipped=1) 2022-11-23T02:22:33.3966003Z 2022-11-23T02:22:33.3966120Z Generating XML reports... 2022-11-23T02:22:33.3966614Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020701.xml 2022-11-23T02:22:33.3966953Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3967353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3967529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3967934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3968125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3968131Z 2022-11-23T02:22:33.3968230Z Running tests... 2022-11-23T02:22:33.3968510Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3968848Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40161 2022-11-23T02:22:33.3969072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40162 2022-11-23T02:22:33.3969352Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3969748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3969920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3970332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3970515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3970765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.3971221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3971380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3971790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3971983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3972233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3972667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3973099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.3973329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.3973554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.3973688Z skip: Test skipped for ROCm (3.652s) 2022-11-23T02:22:33.3973695Z 2022-11-23T02:22:33.3973978Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3974080Z Ran 1 test in 3.653s 2022-11-23T02:22:33.3974163Z 2022-11-23T02:22:33.3974264Z OK (skipped=1) 2022-11-23T02:22:33.3974271Z 2022-11-23T02:22:33.3974389Z Generating XML reports... 2022-11-23T02:22:33.3974889Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020704.xml 2022-11-23T02:22:33.3975224Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3975619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3975795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3976211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3976402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3976409Z 2022-11-23T02:22:33.3976510Z Running tests... 2022-11-23T02:22:33.3976796Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3977058Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:22:33.3977065Z 2022-11-23T02:22:33.3977344Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3977435Z Ran 1 test in 0.002s 2022-11-23T02:22:33.3977453Z 2022-11-23T02:22:33.3977538Z OK (skipped=1) 2022-11-23T02:22:33.3977545Z 2022-11-23T02:22:33.3977661Z Generating XML reports... 2022-11-23T02:22:33.3978157Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020710.xml 2022-11-23T02:22:33.3978501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.3978904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3979086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3979497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3979691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3979698Z 2022-11-23T02:22:33.3979802Z Running tests... 2022-11-23T02:22:33.3980085Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.3980415Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40398 2022-11-23T02:22:33.3980680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40399 2022-11-23T02:22:33.3980959Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.3981366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.3981542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.3981955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.3982149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.3982403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.3982796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4005254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4005761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4005953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4006384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4006828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4007255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4007481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4007704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4007854Z skip: Skipped due to small world size. (3.752s) 2022-11-23T02:22:33.4007867Z 2022-11-23T02:22:33.4008152Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4008249Z Ran 1 test in 3.753s 2022-11-23T02:22:33.4008256Z 2022-11-23T02:22:33.4008345Z OK (skipped=1) 2022-11-23T02:22:33.4008352Z 2022-11-23T02:22:33.4008464Z Generating XML reports... 2022-11-23T02:22:33.4008957Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020713.xml 2022-11-23T02:22:33.4009288Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4009685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4009855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4010259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4010444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4010451Z 2022-11-23T02:22:33.4010544Z Running tests... 2022-11-23T02:22:33.4010818Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4011114Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:22:33.4011122Z 2022-11-23T02:22:33.4011397Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4011488Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4011494Z 2022-11-23T02:22:33.4011585Z OK (skipped=1) 2022-11-23T02:22:33.4011591Z 2022-11-23T02:22:33.4011699Z Generating XML reports... 2022-11-23T02:22:33.4012182Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020720.xml 2022-11-23T02:22:33.4012513Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4012977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4013142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4013546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4013729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4013736Z 2022-11-23T02:22:33.4013828Z Running tests... 2022-11-23T02:22:33.4014103Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4014402Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:22:33.4014409Z 2022-11-23T02:22:33.4014678Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4014775Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4014782Z 2022-11-23T02:22:33.4014870Z OK (skipped=1) 2022-11-23T02:22:33.4014876Z 2022-11-23T02:22:33.4014982Z Generating XML reports... 2022-11-23T02:22:33.4015464Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020723.xml 2022-11-23T02:22:33.4015840Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4016249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4016426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4016840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4017031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4017039Z 2022-11-23T02:22:33.4017125Z Running tests... 2022-11-23T02:22:33.4017412Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4018445Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.517s) 2022-11-23T02:22:33.4018466Z 2022-11-23T02:22:33.4018746Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4018836Z Ran 1 test in 0.517s 2022-11-23T02:22:33.4018843Z 2022-11-23T02:22:33.4018940Z OK (skipped=1) 2022-11-23T02:22:33.4018946Z 2022-11-23T02:22:33.4019060Z Generating XML reports... 2022-11-23T02:22:33.4019550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020726.xml 2022-11-23T02:22:33.4019891Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4020290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4020462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4020876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4021067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4021073Z 2022-11-23T02:22:33.4021173Z Running tests... 2022-11-23T02:22:33.4021460Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4021825Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40751 2022-11-23T02:22:33.4022048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40752 2022-11-23T02:22:33.4022375Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4022777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4022952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4023363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4023552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4023803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4024200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4024374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4024786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4024962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4025285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4025724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4026157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4026391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4026622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4026989Z STAGE:2022-11-23 02:07:32 40751:40751 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4027348Z STAGE:2022-11-23 02:07:33 40752:40752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4027714Z STAGE:2022-11-23 02:07:33 40752:40752 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4028101Z STAGE:2022-11-23 02:07:33 40752:40752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4028345Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.4028714Z STAGE:2022-11-23 02:07:33 40751:40751 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4029095Z STAGE:2022-11-23 02:07:33 40751:40751 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4029337Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.4029436Z ok (4.452s) 2022-11-23T02:22:33.4029443Z 2022-11-23T02:22:33.4029734Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4029839Z Ran 1 test in 4.452s 2022-11-23T02:22:33.4029845Z 2022-11-23T02:22:33.4029929Z OK 2022-11-23T02:22:33.4029935Z 2022-11-23T02:22:33.4030051Z Generating XML reports... 2022-11-23T02:22:33.4030549Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020730.xml 2022-11-23T02:22:33.4030884Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4031286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4031461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4031861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4032096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4032103Z 2022-11-23T02:22:33.4032199Z Running tests... 2022-11-23T02:22:33.4032485Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4032798Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:22:33.4032806Z 2022-11-23T02:22:33.4033081Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4033183Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4033190Z 2022-11-23T02:22:33.4033288Z OK (skipped=1) 2022-11-23T02:22:33.4033294Z 2022-11-23T02:22:33.4033407Z Generating XML reports... 2022-11-23T02:22:33.4033897Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020737.xml 2022-11-23T02:22:33.4034234Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4034633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4034805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4035253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4035445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4035451Z 2022-11-23T02:22:33.4035547Z Running tests... 2022-11-23T02:22:33.4035829Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4036875Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.529s) 2022-11-23T02:22:33.4036886Z 2022-11-23T02:22:33.4037163Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4037264Z Ran 1 test in 0.530s 2022-11-23T02:22:33.4037271Z 2022-11-23T02:22:33.4037368Z OK (skipped=1) 2022-11-23T02:22:33.4037374Z 2022-11-23T02:22:33.4037495Z Generating XML reports... 2022-11-23T02:22:33.4037982Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020741.xml 2022-11-23T02:22:33.4038317Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4038711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4038886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4039280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4039471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4039478Z 2022-11-23T02:22:33.4039577Z Running tests... 2022-11-23T02:22:33.4039857Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4040254Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:22:33.4040262Z 2022-11-23T02:22:33.4040539Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4040641Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4040648Z 2022-11-23T02:22:33.4040748Z OK (skipped=1) 2022-11-23T02:22:33.4040754Z 2022-11-23T02:22:33.4040872Z Generating XML reports... 2022-11-23T02:22:33.4041367Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020744.xml 2022-11-23T02:22:33.4041766Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4042166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4042341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4042752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4042944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4042951Z 2022-11-23T02:22:33.4043053Z Running tests... 2022-11-23T02:22:33.4043330Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4043692Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41121 2022-11-23T02:22:33.4043913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41122 2022-11-23T02:22:33.4044376Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4044779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4045013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4045427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4045606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4045855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4046259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4046429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4046840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4047030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4047281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4047714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4048146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4048378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4048614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4048777Z skip: Skipped due to small world size. (3.751s) 2022-11-23T02:22:33.4048787Z 2022-11-23T02:22:33.4049076Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4049179Z Ran 1 test in 3.751s 2022-11-23T02:22:33.4049186Z 2022-11-23T02:22:33.4049285Z OK (skipped=1) 2022-11-23T02:22:33.4049291Z 2022-11-23T02:22:33.4049408Z Generating XML reports... 2022-11-23T02:22:33.4049904Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020747.xml 2022-11-23T02:22:33.4050240Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4050641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4050812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4051216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4051452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4051459Z 2022-11-23T02:22:33.4051555Z Running tests... 2022-11-23T02:22:33.4051826Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4052124Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:22:33.4052131Z 2022-11-23T02:22:33.4052412Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4052516Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4052523Z 2022-11-23T02:22:33.4052620Z OK (skipped=1) 2022-11-23T02:22:33.4052627Z 2022-11-23T02:22:33.4052740Z Generating XML reports... 2022-11-23T02:22:33.4053238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020754.xml 2022-11-23T02:22:33.4053579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4053978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4054149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4054600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4054792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4054799Z 2022-11-23T02:22:33.4054899Z Running tests... 2022-11-23T02:22:33.4055186Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4055499Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:22:33.4055506Z 2022-11-23T02:22:33.4055785Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4055888Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4055898Z 2022-11-23T02:22:33.4055996Z OK (skipped=1) 2022-11-23T02:22:33.4056002Z 2022-11-23T02:22:33.4056119Z Generating XML reports... 2022-11-23T02:22:33.4056607Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020757.xml 2022-11-23T02:22:33.4056947Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4057347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4057519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4057918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4058110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4058117Z 2022-11-23T02:22:33.4058234Z Running tests... 2022-11-23T02:22:33.4058517Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4058874Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41416 2022-11-23T02:22:33.4059096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41417 2022-11-23T02:22:33.4059374Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4059769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4059948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4060359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4060549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4060847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4061246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4061420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4061833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4062024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4062277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4062712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4063141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4063374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4063610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4063701Z ok (4.445s) 2022-11-23T02:22:33.4063709Z 2022-11-23T02:22:33.4064037Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4064130Z Ran 1 test in 4.445s 2022-11-23T02:22:33.4064136Z 2022-11-23T02:22:33.4064219Z OK 2022-11-23T02:22:33.4064225Z 2022-11-23T02:22:33.4064341Z Generating XML reports... 2022-11-23T02:22:33.4064840Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020800.xml 2022-11-23T02:22:33.4065178Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4065577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4065755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4066164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4066355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4066365Z 2022-11-23T02:22:33.4066463Z Running tests... 2022-11-23T02:22:33.4066750Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4067114Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41606 2022-11-23T02:22:33.4067339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41607 2022-11-23T02:22:33.4067619Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4068018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4068191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4068599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4068794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4069045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4069441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4069615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4070022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4070199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4070496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4070935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4071372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4071607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4071837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4071930Z ok (4.455s) 2022-11-23T02:22:33.4071937Z 2022-11-23T02:22:33.4072226Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4072332Z Ran 1 test in 4.456s 2022-11-23T02:22:33.4072339Z 2022-11-23T02:22:33.4072422Z OK 2022-11-23T02:22:33.4072431Z 2022-11-23T02:22:33.4072549Z Generating XML reports... 2022-11-23T02:22:33.4073041Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020808.xml 2022-11-23T02:22:33.4073376Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4073820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4073994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4074400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4074589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4074595Z 2022-11-23T02:22:33.4074694Z Running tests... 2022-11-23T02:22:33.4074977Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4075293Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:22:33.4075299Z 2022-11-23T02:22:33.4075573Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4075674Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4075684Z 2022-11-23T02:22:33.4075770Z OK (skipped=1) 2022-11-23T02:22:33.4075792Z 2022-11-23T02:22:33.4075895Z Generating XML reports... 2022-11-23T02:22:33.4076381Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020815.xml 2022-11-23T02:22:33.4076716Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4077116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4077290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4077702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4077895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4077902Z 2022-11-23T02:22:33.4078002Z Running tests... 2022-11-23T02:22:33.4078287Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4078660Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41854 2022-11-23T02:22:33.4078877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41855 2022-11-23T02:22:33.4079154Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4079551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4079771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4080255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4080445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4080699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4081099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4081273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4081685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4081876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4082129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4082552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4083034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4083270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4083514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.4083738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4083975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.4084404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4084831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4084923Z ok (4.541s) 2022-11-23T02:22:33.4084930Z 2022-11-23T02:22:33.4085217Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4085326Z Ran 1 test in 4.541s 2022-11-23T02:22:33.4085333Z 2022-11-23T02:22:33.4085415Z OK 2022-11-23T02:22:33.4085421Z 2022-11-23T02:22:33.4085535Z Generating XML reports... 2022-11-23T02:22:33.4086029Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020818.xml 2022-11-23T02:22:33.4086368Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4086769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4086942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4087358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4087551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4087558Z 2022-11-23T02:22:33.4087658Z Running tests... 2022-11-23T02:22:33.4087950Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4088259Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:22:33.4088267Z 2022-11-23T02:22:33.4088547Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4088636Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4088657Z 2022-11-23T02:22:33.4088742Z OK (skipped=1) 2022-11-23T02:22:33.4088759Z 2022-11-23T02:22:33.4088861Z Generating XML reports... 2022-11-23T02:22:33.4089357Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020826.xml 2022-11-23T02:22:33.4089753Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4090153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4090330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4090738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4090925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4090932Z 2022-11-23T02:22:33.4091029Z Running tests... 2022-11-23T02:22:33.4091312Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4091671Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42104 2022-11-23T02:22:33.4091896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42105 2022-11-23T02:22:33.4092170Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4092606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4092781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4093189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4093380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4093632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4094030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4094205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4094613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4094807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4095041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4095477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4095906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4096138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4096366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4096527Z skip: Skipped due to small world size. (3.739s) 2022-11-23T02:22:33.4096534Z 2022-11-23T02:22:33.4096821Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4096927Z Ran 1 test in 3.740s 2022-11-23T02:22:33.4096933Z 2022-11-23T02:22:33.4097031Z OK (skipped=1) 2022-11-23T02:22:33.4097040Z 2022-11-23T02:22:33.4097156Z Generating XML reports... 2022-11-23T02:22:33.4097645Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020829.xml 2022-11-23T02:22:33.4097980Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4098376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4098544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4098954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4099214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4099221Z 2022-11-23T02:22:33.4099320Z Running tests... 2022-11-23T02:22:33.4099607Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4099949Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42283 2022-11-23T02:22:33.4100173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42284 2022-11-23T02:22:33.4100453Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4100851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4101024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4101424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4101618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4101915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4102321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4102494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4102902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4103092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4103344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4103782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4104209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4104443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4104673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4104913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.4105155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.4105578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4105999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4106093Z ok (4.542s) 2022-11-23T02:22:33.4106100Z 2022-11-23T02:22:33.4106384Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4106487Z Ran 1 test in 4.543s 2022-11-23T02:22:33.4106494Z 2022-11-23T02:22:33.4106576Z OK 2022-11-23T02:22:33.4106586Z 2022-11-23T02:22:33.4106702Z Generating XML reports... 2022-11-23T02:22:33.4107202Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020835.xml 2022-11-23T02:22:33.4107539Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4107924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4108095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4108502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4108751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4108758Z 2022-11-23T02:22:33.4108860Z Running tests... 2022-11-23T02:22:33.4109148Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4109621Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:22:33.4109628Z 2022-11-23T02:22:33.4109909Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4110009Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4110016Z 2022-11-23T02:22:33.4110112Z OK (skipped=1) 2022-11-23T02:22:33.4110118Z 2022-11-23T02:22:33.4110235Z Generating XML reports... 2022-11-23T02:22:33.4110721Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020843.xml 2022-11-23T02:22:33.4111061Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4111457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4111628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4112074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4112267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4112273Z 2022-11-23T02:22:33.4112372Z Running tests... 2022-11-23T02:22:33.4112655Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4113108Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:22:33.4113116Z 2022-11-23T02:22:33.4113398Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4113504Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4113511Z 2022-11-23T02:22:33.4113609Z OK (skipped=1) 2022-11-23T02:22:33.4113615Z 2022-11-23T02:22:33.4113717Z Generating XML reports... 2022-11-23T02:22:33.4114214Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020846.xml 2022-11-23T02:22:33.4114554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4114952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4115129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4115537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4115732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4115742Z 2022-11-23T02:22:33.4115843Z Running tests... 2022-11-23T02:22:33.4116124Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4116383Z test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.001s) 2022-11-23T02:22:33.4116390Z 2022-11-23T02:22:33.4116671Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4116775Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4116782Z 2022-11-23T02:22:33.4116875Z OK (skipped=1) 2022-11-23T02:22:33.4116880Z 2022-11-23T02:22:33.4116992Z Generating XML reports... 2022-11-23T02:22:33.4117483Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020849.xml 2022-11-23T02:22:33.4117823Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4118221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4118439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4118852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4119048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4119055Z 2022-11-23T02:22:33.4119154Z Running tests... 2022-11-23T02:22:33.4119435Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4119746Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42656 2022-11-23T02:22:33.4119967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42657 2022-11-23T02:22:33.4120316Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4120720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4120890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4121296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4121544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4121793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4122194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4122367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4122772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4122960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4123216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4123646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4124073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4124309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4124542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4124634Z ok (5.146s) 2022-11-23T02:22:33.4124641Z 2022-11-23T02:22:33.4124926Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4125030Z Ran 1 test in 5.147s 2022-11-23T02:22:33.4125037Z 2022-11-23T02:22:33.4125120Z OK 2022-11-23T02:22:33.4125130Z 2022-11-23T02:22:33.4125246Z Generating XML reports... 2022-11-23T02:22:33.4125740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020852.xml 2022-11-23T02:22:33.4126064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4126466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4126638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4127049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4127238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4127244Z 2022-11-23T02:22:33.4127343Z Running tests... 2022-11-23T02:22:33.4127630Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4127957Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.001s) 2022-11-23T02:22:33.4127965Z 2022-11-23T02:22:33.4128246Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4128348Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4128355Z 2022-11-23T02:22:33.4128455Z OK (skipped=1) 2022-11-23T02:22:33.4128461Z 2022-11-23T02:22:33.4128576Z Generating XML reports... 2022-11-23T02:22:33.4129066Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020900.xml 2022-11-23T02:22:33.4129402Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4129797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4129969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4130383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4130580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4130587Z 2022-11-23T02:22:33.4130685Z Running tests... 2022-11-23T02:22:33.4131011Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4131354Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42900 2022-11-23T02:22:33.4131578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42901 2022-11-23T02:22:33.4131852Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4132238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4132411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4132818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4133010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4133266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4133669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4133847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4134254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4134444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4134697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4135132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4135566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4135801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4136033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4136189Z skip: Skipped due to small world size. (3.797s) 2022-11-23T02:22:33.4136196Z 2022-11-23T02:22:33.4136483Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4136585Z Ran 1 test in 3.797s 2022-11-23T02:22:33.4136592Z 2022-11-23T02:22:33.4136689Z OK (skipped=1) 2022-11-23T02:22:33.4136696Z 2022-11-23T02:22:33.4136811Z Generating XML reports... 2022-11-23T02:22:33.4137307Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020903.xml 2022-11-23T02:22:33.4137689Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4138090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4138255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4138665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4138856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4138863Z 2022-11-23T02:22:33.4138962Z Running tests... 2022-11-23T02:22:33.4139246Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4139509Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.001s) 2022-11-23T02:22:33.4139519Z 2022-11-23T02:22:33.4139800Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4139903Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4139910Z 2022-11-23T02:22:33.4140009Z OK (skipped=1) 2022-11-23T02:22:33.4140016Z 2022-11-23T02:22:33.4140129Z Generating XML reports... 2022-11-23T02:22:33.4140664Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020910.xml 2022-11-23T02:22:33.4141002Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4141403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4141573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4141981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4142173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4142180Z 2022-11-23T02:22:33.4142280Z Running tests... 2022-11-23T02:22:33.4142564Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4142903Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43137 2022-11-23T02:22:33.4143123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43138 2022-11-23T02:22:33.4143394Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4143956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4144128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4144526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4144721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4144967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4145366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4145540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4145948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4146139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4146391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4146826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4147314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4147547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4147781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4147937Z skip: Skipped due to small world size. (3.841s) 2022-11-23T02:22:33.4147944Z 2022-11-23T02:22:33.4148230Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4148335Z Ran 1 test in 3.841s 2022-11-23T02:22:33.4148342Z 2022-11-23T02:22:33.4148445Z OK (skipped=1) 2022-11-23T02:22:33.4148451Z 2022-11-23T02:22:33.4148569Z Generating XML reports... 2022-11-23T02:22:33.4149058Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020912.xml 2022-11-23T02:22:33.4149398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4149796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4149969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4150424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4150603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4150622Z 2022-11-23T02:22:33.4150710Z Running tests... 2022-11-23T02:22:33.4150993Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4151287Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.001s) 2022-11-23T02:22:33.4151294Z 2022-11-23T02:22:33.4151578Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4151685Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4151691Z 2022-11-23T02:22:33.4151790Z OK (skipped=1) 2022-11-23T02:22:33.4151797Z 2022-11-23T02:22:33.4151912Z Generating XML reports... 2022-11-23T02:22:33.4152402Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020919.xml 2022-11-23T02:22:33.4152740Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4153137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4153314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4153720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4153909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4153917Z 2022-11-23T02:22:33.4154021Z Running tests... 2022-11-23T02:22:33.4154301Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4154581Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s) 2022-11-23T02:22:33.4154588Z 2022-11-23T02:22:33.4154871Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4154972Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4154979Z 2022-11-23T02:22:33.4155071Z OK (skipped=1) 2022-11-23T02:22:33.4155077Z 2022-11-23T02:22:33.4155188Z Generating XML reports... 2022-11-23T02:22:33.4155675Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020922.xml 2022-11-23T02:22:33.4156010Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4156392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4156609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4157020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4157210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4157220Z 2022-11-23T02:22:33.4157324Z Running tests... 2022-11-23T02:22:33.4157612Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4157900Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.001s) 2022-11-23T02:22:33.4157907Z 2022-11-23T02:22:33.4158189Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4158293Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4158300Z 2022-11-23T02:22:33.4158397Z OK (skipped=1) 2022-11-23T02:22:33.4158403Z 2022-11-23T02:22:33.4158519Z Generating XML reports... 2022-11-23T02:22:33.4159010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020925.xml 2022-11-23T02:22:33.4159348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4159787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4159963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4160452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4160641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4160648Z 2022-11-23T02:22:33.4160749Z Running tests... 2022-11-23T02:22:33.4161034Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4161299Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s) 2022-11-23T02:22:33.4161310Z 2022-11-23T02:22:33.4161593Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4161697Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4161703Z 2022-11-23T02:22:33.4161800Z OK (skipped=1) 2022-11-23T02:22:33.4161807Z 2022-11-23T02:22:33.4161913Z Generating XML reports... 2022-11-23T02:22:33.4162399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020928.xml 2022-11-23T02:22:33.4162733Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4163133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4163311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4163723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4163917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4163924Z 2022-11-23T02:22:33.4164025Z Running tests... 2022-11-23T02:22:33.4164311Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4164590Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s) 2022-11-23T02:22:33.4164598Z 2022-11-23T02:22:33.4164878Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4165010Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4165016Z 2022-11-23T02:22:33.4165117Z OK (skipped=1) 2022-11-23T02:22:33.4165123Z 2022-11-23T02:22:33.4165242Z Generating XML reports... 2022-11-23T02:22:33.4165730Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020931.xml 2022-11-23T02:22:33.4166069Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4166526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4166698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4167109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4167299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4167306Z 2022-11-23T02:22:33.4167403Z Running tests... 2022-11-23T02:22:33.4167682Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4168025Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43606 2022-11-23T02:22:33.4168242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43607 2022-11-23T02:22:33.4168524Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4168923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4169169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4169583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4169777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4170029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4170429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4170602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4171014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4171202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4171453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4171893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4172327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4172556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4172786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4173031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.4173275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.4173709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4174136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4174382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:22:33.4174623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:22:33.4175047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:22:33.4175470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:22:33.4175596Z ok (3.843s) 2022-11-23T02:22:33.4175618Z 2022-11-23T02:22:33.4175895Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4175998Z Ran 1 test in 3.844s 2022-11-23T02:22:33.4176005Z 2022-11-23T02:22:33.4176090Z OK 2022-11-23T02:22:33.4176097Z 2022-11-23T02:22:33.4176213Z Generating XML reports... 2022-11-23T02:22:33.4176716Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020934.xml 2022-11-23T02:22:33.4177053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4177456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4177633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4178043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4178240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4178247Z 2022-11-23T02:22:33.4178347Z Running tests... 2022-11-23T02:22:33.4178635Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4179018Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43793 2022-11-23T02:22:33.4179246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43794 2022-11-23T02:22:33.4179529Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4179932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4180106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4180515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4180705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4180950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4181350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4181509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4181915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4182101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4182352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4182783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4183216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4183448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4183685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4183782Z ok (4.443s) 2022-11-23T02:22:33.4183788Z 2022-11-23T02:22:33.4184077Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4184181Z Ran 1 test in 4.443s 2022-11-23T02:22:33.4184188Z 2022-11-23T02:22:33.4184271Z OK 2022-11-23T02:22:33.4184278Z 2022-11-23T02:22:33.4184392Z Generating XML reports... 2022-11-23T02:22:33.4184888Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020941.xml 2022-11-23T02:22:33.4185223Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4185833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4186006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4186421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4186612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4186619Z 2022-11-23T02:22:33.4186716Z Running tests... 2022-11-23T02:22:33.4187001Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4187349Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43983 2022-11-23T02:22:33.4187560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43984 2022-11-23T02:22:33.4187840Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4188238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4188456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4188869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4189060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4189309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4189704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4189879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4190292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4190487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4190738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4191173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4191606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4191836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4192068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4192225Z skip: Skipped due to small world size. (3.750s) 2022-11-23T02:22:33.4192232Z 2022-11-23T02:22:33.4192518Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4192620Z Ran 1 test in 3.751s 2022-11-23T02:22:33.4192627Z 2022-11-23T02:22:33.4192722Z OK (skipped=1) 2022-11-23T02:22:33.4192729Z 2022-11-23T02:22:33.4192845Z Generating XML reports... 2022-11-23T02:22:33.4193343Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020948.xml 2022-11-23T02:22:33.4193682Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4194068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4194242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4194649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4194839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4194892Z 2022-11-23T02:22:33.4194995Z Running tests... 2022-11-23T02:22:33.4195281Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4195625Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44162 2022-11-23T02:22:33.4195849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44163 2022-11-23T02:22:33.4196125Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4196528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4196703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4197111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4197303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4197553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4197993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4198172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4198586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4198777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4199026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4199457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4199890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4200117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4200417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4200500Z ok (4.143s) 2022-11-23T02:22:33.4200522Z 2022-11-23T02:22:33.4200797Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4200899Z Ran 1 test in 4.144s 2022-11-23T02:22:33.4200905Z 2022-11-23T02:22:33.4200982Z OK 2022-11-23T02:22:33.4200988Z 2022-11-23T02:22:33.4201103Z Generating XML reports... 2022-11-23T02:22:33.4201596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020955.xml 2022-11-23T02:22:33.4201937Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4202340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4202517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4202929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4203121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4203128Z 2022-11-23T02:22:33.4203230Z Running tests... 2022-11-23T02:22:33.4203518Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4203862Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44341 2022-11-23T02:22:33.4204080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44342 2022-11-23T02:22:33.4204356Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4204815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4204987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4205399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4205590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4205842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4206235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4206393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4206796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4206989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4207234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4207717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4208155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4208388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4208618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4208715Z ok (3.844s) 2022-11-23T02:22:33.4208723Z 2022-11-23T02:22:33.4209011Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4209126Z Ran 1 test in 3.844s 2022-11-23T02:22:33.4209133Z 2022-11-23T02:22:33.4209216Z OK 2022-11-23T02:22:33.4209223Z 2022-11-23T02:22:33.4209341Z Generating XML reports... 2022-11-23T02:22:33.4209839Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021002.xml 2022-11-23T02:22:33.4210180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4210579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4210753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4211162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4211341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4211362Z 2022-11-23T02:22:33.4211447Z Running tests... 2022-11-23T02:22:33.4211737Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4212099Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44520 2022-11-23T02:22:33.4212320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44521 2022-11-23T02:22:33.4212598Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4212991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4213160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4213560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4213750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4214048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4214445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4214621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4215026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4215219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4215468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4215903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4216330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4216564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4216794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4216888Z ok (4.857s) 2022-11-23T02:22:33.4216895Z 2022-11-23T02:22:33.4217226Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4217319Z Ran 1 test in 4.858s 2022-11-23T02:22:33.4217338Z 2022-11-23T02:22:33.4217408Z OK 2022-11-23T02:22:33.4217414Z 2022-11-23T02:22:33.4217530Z Generating XML reports... 2022-11-23T02:22:33.4218026Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021009.xml 2022-11-23T02:22:33.4218367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4218763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4218942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4219352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4219546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4219552Z 2022-11-23T02:22:33.4219652Z Running tests... 2022-11-23T02:22:33.4219936Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4220282Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44706 2022-11-23T02:22:33.4220500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44707 2022-11-23T02:22:33.4220778Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4221178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4221349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4221756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4221952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4222207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4222607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4222780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4223191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4223383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4223665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4224101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4224537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4224775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4225005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4225098Z ok (4.543s) 2022-11-23T02:22:33.4225105Z 2022-11-23T02:22:33.4225393Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4225499Z Ran 1 test in 4.543s 2022-11-23T02:22:33.4225506Z 2022-11-23T02:22:33.4225591Z OK 2022-11-23T02:22:33.4225600Z 2022-11-23T02:22:33.4225714Z Generating XML reports... 2022-11-23T02:22:33.4226209Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021017.xml 2022-11-23T02:22:33.4226544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4226993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4227166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4227583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4227779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4227785Z 2022-11-23T02:22:33.4227889Z Running tests... 2022-11-23T02:22:33.4228176Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4228533Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44892 2022-11-23T02:22:33.4228759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44893 2022-11-23T02:22:33.4229045Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4229443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4229604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4230010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4230202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4230449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4230852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4231026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4231440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4231628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4231879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4232314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4232747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4232983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4233261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4233353Z ok (3.839s) 2022-11-23T02:22:33.4233360Z 2022-11-23T02:22:33.4233645Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4233756Z Ran 1 test in 3.839s 2022-11-23T02:22:33.4233763Z 2022-11-23T02:22:33.4233849Z OK 2022-11-23T02:22:33.4233855Z 2022-11-23T02:22:33.4233974Z Generating XML reports... 2022-11-23T02:22:33.4234470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021024.xml 2022-11-23T02:22:33.4234810Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4235209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4235386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4235784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4235974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4235997Z 2022-11-23T02:22:33.4236087Z Running tests... 2022-11-23T02:22:33.4236417Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4236683Z test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.4236690Z 2022-11-23T02:22:33.4236974Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4237081Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4237087Z 2022-11-23T02:22:33.4237186Z OK (skipped=1) 2022-11-23T02:22:33.4237193Z 2022-11-23T02:22:33.4237308Z Generating XML reports... 2022-11-23T02:22:33.4237798Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021031.xml 2022-11-23T02:22:33.4238134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4238534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4238712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4239118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4239308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4239314Z 2022-11-23T02:22:33.4239411Z Running tests... 2022-11-23T02:22:33.4239694Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4240759Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.544s) 2022-11-23T02:22:33.4240770Z 2022-11-23T02:22:33.4241057Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4241164Z Ran 1 test in 0.544s 2022-11-23T02:22:33.4241171Z 2022-11-23T02:22:33.4241275Z OK (skipped=1) 2022-11-23T02:22:33.4241281Z 2022-11-23T02:22:33.4241398Z Generating XML reports... 2022-11-23T02:22:33.4241892Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021034.xml 2022-11-23T02:22:33.4242233Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4242630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4242805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4243287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4243474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4243480Z 2022-11-23T02:22:33.4243580Z Running tests... 2022-11-23T02:22:33.4243865Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4244141Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.4244148Z 2022-11-23T02:22:33.4244426Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4244525Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4244531Z 2022-11-23T02:22:33.4244627Z OK (skipped=1) 2022-11-23T02:22:33.4244633Z 2022-11-23T02:22:33.4244742Z Generating XML reports... 2022-11-23T02:22:33.4245230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021038.xml 2022-11-23T02:22:33.4245566Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4245965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4246189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4246609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4246806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4246812Z 2022-11-23T02:22:33.4246916Z Running tests... 2022-11-23T02:22:33.4247202Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4247477Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.4247487Z 2022-11-23T02:22:33.4247768Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4247877Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4247883Z 2022-11-23T02:22:33.4247983Z OK (skipped=1) 2022-11-23T02:22:33.4247988Z 2022-11-23T02:22:33.4248101Z Generating XML reports... 2022-11-23T02:22:33.4248739Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021041.xml 2022-11-23T02:22:33.4249082Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4249477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4249652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4250060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4250254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4250262Z 2022-11-23T02:22:33.4250360Z Running tests... 2022-11-23T02:22:33.4250636Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4250916Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.001s) 2022-11-23T02:22:33.4250923Z 2022-11-23T02:22:33.4251198Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4251297Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4251304Z 2022-11-23T02:22:33.4251398Z OK (skipped=1) 2022-11-23T02:22:33.4251404Z 2022-11-23T02:22:33.4251525Z Generating XML reports... 2022-11-23T02:22:33.4252009Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021044.xml 2022-11-23T02:22:33.4252342Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4252788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4252963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4253367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4253563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4253570Z 2022-11-23T02:22:33.4253670Z Running tests... 2022-11-23T02:22:33.4253950Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4254946Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:22:33.4254957Z 2022-11-23T02:22:33.4255240Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4255343Z Ran 1 test in 0.510s 2022-11-23T02:22:33.4255349Z 2022-11-23T02:22:33.4255447Z OK (skipped=1) 2022-11-23T02:22:33.4255453Z 2022-11-23T02:22:33.4255568Z Generating XML reports... 2022-11-23T02:22:33.4256090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021047.xml 2022-11-23T02:22:33.4256433Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4256829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4257003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4257416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4257609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4257616Z 2022-11-23T02:22:33.4257715Z Running tests... 2022-11-23T02:22:33.4258004Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4258563Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.001s) 2022-11-23T02:22:33.4258571Z 2022-11-23T02:22:33.4258853Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4258961Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4258968Z 2022-11-23T02:22:33.4259067Z OK (skipped=1) 2022-11-23T02:22:33.4259074Z 2022-11-23T02:22:33.4259191Z Generating XML reports... 2022-11-23T02:22:33.4259677Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021050.xml 2022-11-23T02:22:33.4260018Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4260418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4260591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4261004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4261194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4261200Z 2022-11-23T02:22:33.4261299Z Running tests... 2022-11-23T02:22:33.4261584Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4262143Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:22:33.4262151Z 2022-11-23T02:22:33.4262481Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4262579Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4262587Z 2022-11-23T02:22:33.4262670Z OK (skipped=1) 2022-11-23T02:22:33.4262689Z 2022-11-23T02:22:33.4262790Z Generating XML reports... 2022-11-23T02:22:33.4263283Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021053.xml 2022-11-23T02:22:33.4263619Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4265414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4265593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4266003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4266199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4266209Z 2022-11-23T02:22:33.4266309Z Running tests... 2022-11-23T02:22:33.4266589Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4266927Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45535 2022-11-23T02:22:33.4267209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45536 2022-11-23T02:22:33.4267485Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4267883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4268052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4268463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4268651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4268901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4269296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4269469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4269878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4270067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4270321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4270759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4271177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4271412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4271647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4271915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2s7xro2w 2022-11-23T02:22:33.4272195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2s7xro2w/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4272450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ujxzk89 2022-11-23T02:22:33.4272720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ujxzk89/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4272957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4273198Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4273341Z ok (6.647s) 2022-11-23T02:22:33.4273349Z 2022-11-23T02:22:33.4273637Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4273745Z Ran 1 test in 6.648s 2022-11-23T02:22:33.4273752Z 2022-11-23T02:22:33.4273833Z OK 2022-11-23T02:22:33.4273844Z 2022-11-23T02:22:33.4274036Z Generating XML reports... 2022-11-23T02:22:33.4274530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021056.xml 2022-11-23T02:22:33.4274865Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4275255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4275431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4275836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4276030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4276037Z 2022-11-23T02:22:33.4276136Z Running tests... 2022-11-23T02:22:33.4276405Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4276794Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45731 2022-11-23T02:22:33.4277019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45732 2022-11-23T02:22:33.4277298Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4277699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4277875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4278286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4278475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4278729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4279118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4279293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4279703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4279891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4280217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4280654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4281086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4281322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4281552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4281815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplply1yjx 2022-11-23T02:22:33.4282099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplply1yjx/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4282358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkm3zl_2f 2022-11-23T02:22:33.4282630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkm3zl_2f/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4282930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4283169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4283392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4283633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4283728Z ok (6.448s) 2022-11-23T02:22:33.4283735Z 2022-11-23T02:22:33.4284027Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4284133Z Ran 1 test in 6.448s 2022-11-23T02:22:33.4284140Z 2022-11-23T02:22:33.4284223Z OK 2022-11-23T02:22:33.4284230Z 2022-11-23T02:22:33.4284348Z Generating XML reports... 2022-11-23T02:22:33.4284841Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021106.xml 2022-11-23T02:22:33.4285178Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4285576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4285746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4286208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4286400Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4286406Z 2022-11-23T02:22:33.4286506Z Running tests... 2022-11-23T02:22:33.4286789Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4287791Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.521s) 2022-11-23T02:22:33.4287802Z 2022-11-23T02:22:33.4288082Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4288192Z Ran 1 test in 0.522s 2022-11-23T02:22:33.4288199Z 2022-11-23T02:22:33.4288302Z OK (skipped=1) 2022-11-23T02:22:33.4288309Z 2022-11-23T02:22:33.4288423Z Generating XML reports... 2022-11-23T02:22:33.4288917Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021115.xml 2022-11-23T02:22:33.4289257Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4289657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4289830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4290227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4290417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4290424Z 2022-11-23T02:22:33.4290523Z Running tests... 2022-11-23T02:22:33.4290813Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4291840Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.520s) 2022-11-23T02:22:33.4291849Z 2022-11-23T02:22:33.4292129Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4292233Z Ran 1 test in 0.520s 2022-11-23T02:22:33.4292284Z 2022-11-23T02:22:33.4292382Z OK (skipped=1) 2022-11-23T02:22:33.4292387Z 2022-11-23T02:22:33.4292507Z Generating XML reports... 2022-11-23T02:22:33.4292996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021119.xml 2022-11-23T02:22:33.4293335Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4293730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4293904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4294312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4294502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4294509Z 2022-11-23T02:22:33.4294607Z Running tests... 2022-11-23T02:22:33.4294888Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4295404Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.003s) 2022-11-23T02:22:33.4295411Z 2022-11-23T02:22:33.4295734Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4295842Z Ran 1 test in 0.003s 2022-11-23T02:22:33.4295848Z 2022-11-23T02:22:33.4295945Z OK (skipped=1) 2022-11-23T02:22:33.4295951Z 2022-11-23T02:22:33.4296074Z Generating XML reports... 2022-11-23T02:22:33.4296579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021122.xml 2022-11-23T02:22:33.4296922Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4297323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4297502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4297900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4298095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4298118Z 2022-11-23T02:22:33.4298207Z Running tests... 2022-11-23T02:22:33.4298496Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4298877Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46101 2022-11-23T02:22:33.4299100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46102 2022-11-23T02:22:33.4299382Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4299779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4299959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4300369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4300565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4300815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4301211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4301389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4301797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4301987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4302284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4302724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4303167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4303405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4303641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4303910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwroy1l0v 2022-11-23T02:22:33.4304199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwroy1l0v/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4304459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4h47dox 2022-11-23T02:22:33.4304724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4h47dox/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4304823Z ok (4.142s) 2022-11-23T02:22:33.4304831Z 2022-11-23T02:22:33.4305119Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4305271Z Ran 1 test in 4.143s 2022-11-23T02:22:33.4305278Z 2022-11-23T02:22:33.4305365Z OK 2022-11-23T02:22:33.4305371Z 2022-11-23T02:22:33.4305488Z Generating XML reports... 2022-11-23T02:22:33.4305985Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021125.xml 2022-11-23T02:22:33.4306323Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4306724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4306901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4307314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4307502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4307509Z 2022-11-23T02:22:33.4307607Z Running tests... 2022-11-23T02:22:33.4307897Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4308228Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46287 2022-11-23T02:22:33.4308445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46288 2022-11-23T02:22:33.4308723Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4309126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4309306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4309717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4309909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4310166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4310554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4310730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4311139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4311330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4311581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4312067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4312502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4312739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4312970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4313233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpct5cgfr8 2022-11-23T02:22:33.4313508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpct5cgfr8/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4313762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsawxsg5d 2022-11-23T02:22:33.4314045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsawxsg5d/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4314285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4314525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4314686Z ok (6.849s) 2022-11-23T02:22:33.4314694Z 2022-11-23T02:22:33.4314985Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4315091Z Ran 1 test in 6.850s 2022-11-23T02:22:33.4315098Z 2022-11-23T02:22:33.4315186Z OK 2022-11-23T02:22:33.4315192Z 2022-11-23T02:22:33.4315317Z Generating XML reports... 2022-11-23T02:22:33.4315814Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021132.xml 2022-11-23T02:22:33.4316139Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4316549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4316727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4317136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4317330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4317337Z 2022-11-23T02:22:33.4317439Z Running tests... 2022-11-23T02:22:33.4317725Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4318235Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.003s) 2022-11-23T02:22:33.4318243Z 2022-11-23T02:22:33.4318528Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4318634Z Ran 1 test in 0.004s 2022-11-23T02:22:33.4318643Z 2022-11-23T02:22:33.4318742Z OK (skipped=1) 2022-11-23T02:22:33.4318748Z 2022-11-23T02:22:33.4318863Z Generating XML reports... 2022-11-23T02:22:33.4319355Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021142.xml 2022-11-23T02:22:33.4319695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4320096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4320339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4320752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4320943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4320950Z 2022-11-23T02:22:33.4321048Z Running tests... 2022-11-23T02:22:33.4321335Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4321895Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s) 2022-11-23T02:22:33.4321902Z 2022-11-23T02:22:33.4322187Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4322294Z Ran 1 test in 0.003s 2022-11-23T02:22:33.4322300Z 2022-11-23T02:22:33.4322403Z OK (skipped=1) 2022-11-23T02:22:33.4322409Z 2022-11-23T02:22:33.4322512Z Generating XML reports... 2022-11-23T02:22:33.4323001Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021145.xml 2022-11-23T02:22:33.4323338Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4323736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4323917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4324330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4324522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4324529Z 2022-11-23T02:22:33.4324680Z Running tests... 2022-11-23T02:22:33.4324966Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4325328Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s) 2022-11-23T02:22:33.4325335Z 2022-11-23T02:22:33.4325621Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4325726Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4325732Z 2022-11-23T02:22:33.4325827Z OK (skipped=1) 2022-11-23T02:22:33.4325833Z 2022-11-23T02:22:33.4325944Z Generating XML reports... 2022-11-23T02:22:33.4326435Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021148.xml 2022-11-23T02:22:33.4326780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4327176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4327352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4327767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4327961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4327968Z 2022-11-23T02:22:33.4328067Z Running tests... 2022-11-23T02:22:33.4328349Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4328783Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.004s) 2022-11-23T02:22:33.4328810Z 2022-11-23T02:22:33.4329075Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4329178Z Ran 1 test in 0.004s 2022-11-23T02:22:33.4329184Z 2022-11-23T02:22:33.4329281Z OK (skipped=1) 2022-11-23T02:22:33.4329288Z 2022-11-23T02:22:33.4329402Z Generating XML reports... 2022-11-23T02:22:33.4329894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021151.xml 2022-11-23T02:22:33.4330233Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4330633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4330808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4331217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4331456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4331463Z 2022-11-23T02:22:33.4331563Z Running tests... 2022-11-23T02:22:33.4331844Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4332331Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:22:33.4332339Z 2022-11-23T02:22:33.4332621Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4332724Z Ran 1 test in 0.003s 2022-11-23T02:22:33.4332731Z 2022-11-23T02:22:33.4332824Z OK (skipped=1) 2022-11-23T02:22:33.4332830Z 2022-11-23T02:22:33.4332942Z Generating XML reports... 2022-11-23T02:22:33.4333436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021154.xml 2022-11-23T02:22:33.4333776Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4334176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4334354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4334809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4335004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4335011Z 2022-11-23T02:22:33.4335098Z Running tests... 2022-11-23T02:22:33.4335383Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4336385Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.508s) 2022-11-23T02:22:33.4336410Z 2022-11-23T02:22:33.4336680Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4336784Z Ran 1 test in 0.509s 2022-11-23T02:22:33.4336791Z 2022-11-23T02:22:33.4336887Z OK (skipped=1) 2022-11-23T02:22:33.4336896Z 2022-11-23T02:22:33.4337016Z Generating XML reports... 2022-11-23T02:22:33.4337503Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021157.xml 2022-11-23T02:22:33.4337841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4338237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4338408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4338816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4339013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4339019Z 2022-11-23T02:22:33.4339113Z Running tests... 2022-11-23T02:22:33.4339394Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4340396Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.524s) 2022-11-23T02:22:33.4340404Z 2022-11-23T02:22:33.4340682Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4340784Z Ran 1 test in 0.524s 2022-11-23T02:22:33.4340790Z 2022-11-23T02:22:33.4340890Z OK (skipped=1) 2022-11-23T02:22:33.4340943Z 2022-11-23T02:22:33.4341061Z Generating XML reports... 2022-11-23T02:22:33.4341556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021200.xml 2022-11-23T02:22:33.4341891Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4342291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4342462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4342873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4343066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4343073Z 2022-11-23T02:22:33.4343174Z Running tests... 2022-11-23T02:22:33.4343454Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4343808Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46889 2022-11-23T02:22:33.4344030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46890 2022-11-23T02:22:33.4344351Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4344752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4344933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4345338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4345532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4345781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4346181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4346355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4346761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4346954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4347204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4347640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4348072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4348468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4348719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.4348946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4349187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.4349621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4350046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4350315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxczv6du4 2022-11-23T02:22:33.4350598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxczv6du4/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4350911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03kqkp0a 2022-11-23T02:22:33.4351172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03kqkp0a/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4351414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4351658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4351894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4352129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4352221Z ok (7.064s) 2022-11-23T02:22:33.4352228Z 2022-11-23T02:22:33.4352517Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4352622Z Ran 1 test in 7.064s 2022-11-23T02:22:33.4352629Z 2022-11-23T02:22:33.4352713Z OK 2022-11-23T02:22:33.4352719Z 2022-11-23T02:22:33.4352836Z Generating XML reports... 2022-11-23T02:22:33.4353332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021204.xml 2022-11-23T02:22:33.4353671Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4354117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4354295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4354708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4354900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4354907Z 2022-11-23T02:22:33.4355007Z Running tests... 2022-11-23T02:22:33.4355291Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4355642Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47087 2022-11-23T02:22:33.4355868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47088 2022-11-23T02:22:33.4356150Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4356548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4356708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4357117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4357304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4357553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4357951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4358125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4358536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4358723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4358969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4366695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4367235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4367480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4367937Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:22:33.4368166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4368450Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:22:33.4368721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphuang481 2022-11-23T02:22:33.4368997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphuang481/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4369254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lyolqt2 2022-11-23T02:22:33.4369523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lyolqt2/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4369759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4369999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4370235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4370473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4370817Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:22:33.4371096Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:22:33.4371371Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:22:33.4371648Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:22:33.4371877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4372105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4372330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4372567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4372847Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:22:33.4373121Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:22:33.4373400Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:22:33.4373674Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:22:33.4373905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4374141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4374376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4374611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4374699Z ok (7.050s) 2022-11-23T02:22:33.4374708Z 2022-11-23T02:22:33.4375005Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4375110Z Ran 1 test in 7.051s 2022-11-23T02:22:33.4375118Z 2022-11-23T02:22:33.4375195Z OK 2022-11-23T02:22:33.4375200Z 2022-11-23T02:22:33.4375312Z Generating XML reports... 2022-11-23T02:22:33.4375811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021214.xml 2022-11-23T02:22:33.4376153Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4376608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4376786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4377200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4377392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4377399Z 2022-11-23T02:22:33.4377487Z Running tests... 2022-11-23T02:22:33.4377776Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4378791Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:22:33.4378815Z 2022-11-23T02:22:33.4379086Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4379188Z Ran 1 test in 0.509s 2022-11-23T02:22:33.4379195Z 2022-11-23T02:22:33.4379291Z OK (skipped=1) 2022-11-23T02:22:33.4379297Z 2022-11-23T02:22:33.4379496Z Generating XML reports... 2022-11-23T02:22:33.4380000Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021224.xml 2022-11-23T02:22:33.4380338Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4380743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4380919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4381335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4381535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4381542Z 2022-11-23T02:22:33.4381641Z Running tests... 2022-11-23T02:22:33.4381930Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4382283Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47341 2022-11-23T02:22:33.4382511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47342 2022-11-23T02:22:33.4382791Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4383189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4383365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4383777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4383971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4384224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4384631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4384806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4385211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4385388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4385641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4386079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4386565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4386800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4387391Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:22:33.4387629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4388209Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:22:33.4388478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnupftc6h 2022-11-23T02:22:33.4388806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnupftc6h/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4389070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotentw12 2022-11-23T02:22:33.4389342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotentw12/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4389583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4389823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4390106Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:22:33.4390391Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:22:33.4390702Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:22:33.4391048Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:22:33.4391355Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:22:33.4391692Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:22:33.4392028Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:22:33.4392354Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:22:33.4392594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4392835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4392925Z ok (6.560s) 2022-11-23T02:22:33.4392932Z 2022-11-23T02:22:33.4393232Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4393340Z Ran 1 test in 6.560s 2022-11-23T02:22:33.4393347Z 2022-11-23T02:22:33.4393431Z OK 2022-11-23T02:22:33.4393438Z 2022-11-23T02:22:33.4393554Z Generating XML reports... 2022-11-23T02:22:33.4394056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021227.xml 2022-11-23T02:22:33.4394384Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4394869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4395042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4395458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4395654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4395661Z 2022-11-23T02:22:33.4395760Z Running tests... 2022-11-23T02:22:33.4396047Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4396466Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4396473Z 2022-11-23T02:22:33.4396761Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4396863Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4396869Z 2022-11-23T02:22:33.4396964Z OK (skipped=1) 2022-11-23T02:22:33.4396971Z 2022-11-23T02:22:33.4397085Z Generating XML reports... 2022-11-23T02:22:33.4397625Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021237.xml 2022-11-23T02:22:33.4397973Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4398379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4398551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4398960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4399152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4399162Z 2022-11-23T02:22:33.4399258Z Running tests... 2022-11-23T02:22:33.4399542Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4399947Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4399955Z 2022-11-23T02:22:33.4400328Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4400434Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4400440Z 2022-11-23T02:22:33.4400542Z OK (skipped=1) 2022-11-23T02:22:33.4400548Z 2022-11-23T02:22:33.4400668Z Generating XML reports... 2022-11-23T02:22:33.4401147Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021240.xml 2022-11-23T02:22:33.4401486Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4401895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4402072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4402482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4402678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4402685Z 2022-11-23T02:22:33.4402784Z Running tests... 2022-11-23T02:22:33.4403062Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4403539Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4403547Z 2022-11-23T02:22:33.4403893Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4403998Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4404005Z 2022-11-23T02:22:33.4404100Z OK (skipped=1) 2022-11-23T02:22:33.4404106Z 2022-11-23T02:22:33.4404223Z Generating XML reports... 2022-11-23T02:22:33.4404717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021243.xml 2022-11-23T02:22:33.4405052Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4405453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4405627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4406043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4406233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4406243Z 2022-11-23T02:22:33.4406344Z Running tests... 2022-11-23T02:22:33.4406626Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4407154Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4407163Z 2022-11-23T02:22:33.4407447Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4407551Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4407557Z 2022-11-23T02:22:33.4407655Z OK (skipped=1) 2022-11-23T02:22:33.4407662Z 2022-11-23T02:22:33.4407779Z Generating XML reports... 2022-11-23T02:22:33.4408255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021246.xml 2022-11-23T02:22:33.4408599Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4409002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4409177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4409591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4409783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4409790Z 2022-11-23T02:22:33.4409891Z Running tests... 2022-11-23T02:22:33.4410170Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4410637Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4410648Z 2022-11-23T02:22:33.4410932Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4411032Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4411038Z 2022-11-23T02:22:33.4411132Z OK (skipped=1) 2022-11-23T02:22:33.4411138Z 2022-11-23T02:22:33.4411250Z Generating XML reports... 2022-11-23T02:22:33.4411745Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021249.xml 2022-11-23T02:22:33.4412087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4412486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4412662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4413074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4413269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4413324Z 2022-11-23T02:22:33.4413426Z Running tests... 2022-11-23T02:22:33.4413709Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4414183Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4414192Z 2022-11-23T02:22:33.4414473Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4414575Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4414581Z 2022-11-23T02:22:33.4414677Z OK (skipped=1) 2022-11-23T02:22:33.4414683Z 2022-11-23T02:22:33.4414786Z Generating XML reports... 2022-11-23T02:22:33.4415277Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021251.xml 2022-11-23T02:22:33.4415621Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4416017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4416191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4416649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4416846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4416853Z 2022-11-23T02:22:33.4416956Z Running tests... 2022-11-23T02:22:33.4417236Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4417704Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4417715Z 2022-11-23T02:22:33.4417995Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4418096Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4418103Z 2022-11-23T02:22:33.4418197Z OK (skipped=1) 2022-11-23T02:22:33.4418203Z 2022-11-23T02:22:33.4418321Z Generating XML reports... 2022-11-23T02:22:33.4418816Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021254.xml 2022-11-23T02:22:33.4419158Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4419562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4419740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4420154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4420350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4420356Z 2022-11-23T02:22:33.4420459Z Running tests... 2022-11-23T02:22:33.4420746Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4421219Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4421227Z 2022-11-23T02:22:33.4421508Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4421612Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4421618Z 2022-11-23T02:22:33.4421714Z OK (skipped=1) 2022-11-23T02:22:33.4421720Z 2022-11-23T02:22:33.4421822Z Generating XML reports... 2022-11-23T02:22:33.4422314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021258.xml 2022-11-23T02:22:33.4422704Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4423102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4423279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4423685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4423880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4423886Z 2022-11-23T02:22:33.4423987Z Running tests... 2022-11-23T02:22:33.4424265Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4424724Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4424734Z 2022-11-23T02:22:33.4425013Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4425118Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4425124Z 2022-11-23T02:22:33.4425225Z OK (skipped=1) 2022-11-23T02:22:33.4425231Z 2022-11-23T02:22:33.4425396Z Generating XML reports... 2022-11-23T02:22:33.4425891Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021300.xml 2022-11-23T02:22:33.4426230Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4426628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4426803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4427213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4427412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4427419Z 2022-11-23T02:22:33.4427515Z Running tests... 2022-11-23T02:22:33.4427799Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4428269Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4428275Z 2022-11-23T02:22:33.4428555Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4428654Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4428660Z 2022-11-23T02:22:33.4428745Z OK (skipped=1) 2022-11-23T02:22:33.4428767Z 2022-11-23T02:22:33.4428870Z Generating XML reports... 2022-11-23T02:22:33.4429359Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021303.xml 2022-11-23T02:22:33.4429701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4430100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4430275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4430687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4430875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4430882Z 2022-11-23T02:22:33.4430981Z Running tests... 2022-11-23T02:22:33.4431263Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4431669Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4431726Z 2022-11-23T02:22:33.4432010Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4432111Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4432118Z 2022-11-23T02:22:33.4432218Z OK (skipped=1) 2022-11-23T02:22:33.4432224Z 2022-11-23T02:22:33.4432347Z Generating XML reports... 2022-11-23T02:22:33.4432831Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021306.xml 2022-11-23T02:22:33.4433169Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4433568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4433741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4434154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4434348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4434355Z 2022-11-23T02:22:33.4434453Z Running tests... 2022-11-23T02:22:33.4434734Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4435178Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:22:33.4435186Z 2022-11-23T02:22:33.4435472Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4435561Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4435579Z 2022-11-23T02:22:33.4435664Z OK (skipped=1) 2022-11-23T02:22:33.4435670Z 2022-11-23T02:22:33.4435788Z Generating XML reports... 2022-11-23T02:22:33.4436278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021309.xml 2022-11-23T02:22:33.4436619Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4437019Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4437198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4437616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4437808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4437815Z 2022-11-23T02:22:33.4437914Z Running tests... 2022-11-23T02:22:33.4438207Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4438681Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:22:33.4438689Z 2022-11-23T02:22:33.4438979Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4439086Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4439092Z 2022-11-23T02:22:33.4439190Z OK (skipped=1) 2022-11-23T02:22:33.4439196Z 2022-11-23T02:22:33.4439312Z Generating XML reports... 2022-11-23T02:22:33.4439811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021312.xml 2022-11-23T02:22:33.4440228Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4440631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4440807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4441213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4441407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4441474Z 2022-11-23T02:22:33.4441575Z Running tests... 2022-11-23T02:22:33.4441848Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4442181Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48291 2022-11-23T02:22:33.4442402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48292 2022-11-23T02:22:33.4442683Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4443079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4443256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4443668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4443863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4444121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4444574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4444752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4445166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4445358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4445615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4446057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4446500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4446734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4446967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4447231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt7l5_iv 2022-11-23T02:22:33.4447508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt7l5_iv/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4447770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0dwz0_m 2022-11-23T02:22:33.4448046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0dwz0_m/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4448126Z ok (6.436s) 2022-11-23T02:22:33.4448147Z 2022-11-23T02:22:33.4448421Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4448529Z Ran 1 test in 6.436s 2022-11-23T02:22:33.4448536Z 2022-11-23T02:22:33.4448614Z OK 2022-11-23T02:22:33.4448621Z 2022-11-23T02:22:33.4448736Z Generating XML reports... 2022-11-23T02:22:33.4449238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021315.xml 2022-11-23T02:22:33.4449575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4449980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4450157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4450570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4450767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4450774Z 2022-11-23T02:22:33.4450925Z Running tests... 2022-11-23T02:22:33.4451209Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4451555Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48480 2022-11-23T02:22:33.4451780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48481 2022-11-23T02:22:33.4452055Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4452455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4452627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4453036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4453229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4453481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4453871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4454088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4454483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4454672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4454913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4455351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4455785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4456016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4456244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4456509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfmctzkh 2022-11-23T02:22:33.4456792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfmctzkh/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4457055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttiarws_ 2022-11-23T02:22:33.4457332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttiarws_/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4457577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4457815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4458257Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:22:33.4458420Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:22:33.4458842Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:22:33.4459001Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:22:33.4459090Z ok (6.460s) 2022-11-23T02:22:33.4459097Z 2022-11-23T02:22:33.4459384Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4459490Z Ran 1 test in 6.461s 2022-11-23T02:22:33.4459497Z 2022-11-23T02:22:33.4459577Z OK 2022-11-23T02:22:33.4459584Z 2022-11-23T02:22:33.4459688Z Generating XML reports... 2022-11-23T02:22:33.4460182Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021324.xml 2022-11-23T02:22:33.4460578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4460978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4461153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4461567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4461757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4461763Z 2022-11-23T02:22:33.4461864Z Running tests... 2022-11-23T02:22:33.4462148Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4462436Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.005s) 2022-11-23T02:22:33.4462443Z 2022-11-23T02:22:33.4462725Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4462834Z Ran 1 test in 0.005s 2022-11-23T02:22:33.4462840Z 2022-11-23T02:22:33.4462942Z OK (skipped=1) 2022-11-23T02:22:33.4462948Z 2022-11-23T02:22:33.4463067Z Generating XML reports... 2022-11-23T02:22:33.4463625Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021334.xml 2022-11-23T02:22:33.4463967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4464371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4464546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4464959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4465153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4465162Z 2022-11-23T02:22:33.4465266Z Running tests... 2022-11-23T02:22:33.4465549Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4465882Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48734 2022-11-23T02:22:33.4466094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48735 2022-11-23T02:22:33.4466371Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4466770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4466947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4467357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4467548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4467805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4468204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4468381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4468791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4468981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4469235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4469671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4470103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4470387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4470618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4470886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3yds_0c 2022-11-23T02:22:33.4471159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3yds_0c/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4471418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcf_vrejf 2022-11-23T02:22:33.4471691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcf_vrejf/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4471927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4472163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4472244Z ok (6.554s) 2022-11-23T02:22:33.4472264Z 2022-11-23T02:22:33.4472541Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4472645Z Ran 1 test in 6.555s 2022-11-23T02:22:33.4472653Z 2022-11-23T02:22:33.4472737Z OK 2022-11-23T02:22:33.4472744Z 2022-11-23T02:22:33.4472915Z Generating XML reports... 2022-11-23T02:22:33.4473416Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021337.xml 2022-11-23T02:22:33.4473757Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4474159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4474334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4474740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4474934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4474941Z 2022-11-23T02:22:33.4475041Z Running tests... 2022-11-23T02:22:33.4475330Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4475847Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:22:33.4475854Z 2022-11-23T02:22:33.4476139Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4476241Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4476247Z 2022-11-23T02:22:33.4476348Z OK (skipped=1) 2022-11-23T02:22:33.4476355Z 2022-11-23T02:22:33.4476470Z Generating XML reports... 2022-11-23T02:22:33.4476957Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021347.xml 2022-11-23T02:22:33.4477295Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4477691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4477862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4478278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4478454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4478471Z 2022-11-23T02:22:33.4478557Z Running tests... 2022-11-23T02:22:33.4478840Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4479335Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:22:33.4479341Z 2022-11-23T02:22:33.4479623Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4479781Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4479788Z 2022-11-23T02:22:33.4479884Z OK (skipped=1) 2022-11-23T02:22:33.4479890Z 2022-11-23T02:22:33.4480008Z Generating XML reports... 2022-11-23T02:22:33.4480568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021350.xml 2022-11-23T02:22:33.4480907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4481302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4481478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4481892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4482081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4482092Z 2022-11-23T02:22:33.4482195Z Running tests... 2022-11-23T02:22:33.4482477Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4483070Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:22:33.4483078Z 2022-11-23T02:22:33.4483363Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4483465Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4483472Z 2022-11-23T02:22:33.4483567Z OK (skipped=1) 2022-11-23T02:22:33.4483573Z 2022-11-23T02:22:33.4483690Z Generating XML reports... 2022-11-23T02:22:33.4484175Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021353.xml 2022-11-23T02:22:33.4484512Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4484919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4485081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4485488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4485682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4485689Z 2022-11-23T02:22:33.4485790Z Running tests... 2022-11-23T02:22:33.4486075Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4486585Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:22:33.4486593Z 2022-11-23T02:22:33.4486871Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4486974Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4486984Z 2022-11-23T02:22:33.4487084Z OK (skipped=1) 2022-11-23T02:22:33.4487090Z 2022-11-23T02:22:33.4487209Z Generating XML reports... 2022-11-23T02:22:33.4487701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021356.xml 2022-11-23T02:22:33.4488046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4488448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4488626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4489039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4489231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4489237Z 2022-11-23T02:22:33.4489337Z Running tests... 2022-11-23T02:22:33.4489620Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4490135Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:22:33.4490142Z 2022-11-23T02:22:33.4490422Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4490523Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4490531Z 2022-11-23T02:22:33.4490628Z OK (skipped=1) 2022-11-23T02:22:33.4490634Z 2022-11-23T02:22:33.4490748Z Generating XML reports... 2022-11-23T02:22:33.4491230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021359.xml 2022-11-23T02:22:33.4491568Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4491962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4492140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4492551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4492737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4492744Z 2022-11-23T02:22:33.4493034Z Running tests... 2022-11-23T02:22:33.4493324Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4493661Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49220 2022-11-23T02:22:33.4493887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49221 2022-11-23T02:22:33.4494168Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4494572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4494752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4495166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4495362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4495617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4496017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4496189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4496604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4496794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4497046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4497484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4497922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4498143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4498374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4498637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1o2i8itv 2022-11-23T02:22:33.4498916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1o2i8itv/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4499176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcir3uf1 2022-11-23T02:22:33.4499499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcir3uf1/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4500343Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:22:33.4501150Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:22:33.4501415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4501698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4501780Z ok (6.359s) 2022-11-23T02:22:33.4501787Z 2022-11-23T02:22:33.4502084Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4502192Z Ran 1 test in 6.360s 2022-11-23T02:22:33.4502198Z 2022-11-23T02:22:33.4502279Z OK 2022-11-23T02:22:33.4502286Z 2022-11-23T02:22:33.4502403Z Generating XML reports... 2022-11-23T02:22:33.4502899Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021402.xml 2022-11-23T02:22:33.4503241Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4503642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4503815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4504225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4504416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4504422Z 2022-11-23T02:22:33.4504524Z Running tests... 2022-11-23T02:22:33.4504810Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4505833Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.529s) 2022-11-23T02:22:33.4505844Z 2022-11-23T02:22:33.4506122Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4506224Z Ran 1 test in 0.530s 2022-11-23T02:22:33.4506231Z 2022-11-23T02:22:33.4506330Z OK (skipped=1) 2022-11-23T02:22:33.4506336Z 2022-11-23T02:22:33.4506453Z Generating XML reports... 2022-11-23T02:22:33.4506942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021411.xml 2022-11-23T02:22:33.4507284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4507690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4507871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4508340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4508536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4508543Z 2022-11-23T02:22:33.4508645Z Running tests... 2022-11-23T02:22:33.4508918Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4509416Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:22:33.4509423Z 2022-11-23T02:22:33.4509709Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4509814Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4509820Z 2022-11-23T02:22:33.4509916Z OK (skipped=1) 2022-11-23T02:22:33.4509921Z 2022-11-23T02:22:33.4510039Z Generating XML reports... 2022-11-23T02:22:33.4510533Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021415.xml 2022-11-23T02:22:33.4510874Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4511272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4511488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4511899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4512091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4512098Z 2022-11-23T02:22:33.4512195Z Running tests... 2022-11-23T02:22:33.4512476Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4512961Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:22:33.4512971Z 2022-11-23T02:22:33.4513253Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4513361Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4513368Z 2022-11-23T02:22:33.4513468Z OK (skipped=1) 2022-11-23T02:22:33.4513474Z 2022-11-23T02:22:33.4513592Z Generating XML reports... 2022-11-23T02:22:33.4514084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021418.xml 2022-11-23T02:22:33.4514418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4514819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4514992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4515385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4515579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4515586Z 2022-11-23T02:22:33.4515687Z Running tests... 2022-11-23T02:22:33.4515977Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4516323Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49590 2022-11-23T02:22:33.4516550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49591 2022-11-23T02:22:33.4516829Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4517222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4517400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4517806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4518047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4518299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4518703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4518875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4519281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4519474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4519730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4520246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4520686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4520922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4521214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4521483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpriu1lidg 2022-11-23T02:22:33.4521758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpriu1lidg/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4522005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mhd1cu_ 2022-11-23T02:22:33.4522278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mhd1cu_/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4522371Z ok (4.434s) 2022-11-23T02:22:33.4522378Z 2022-11-23T02:22:33.4522681Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4522786Z Ran 1 test in 4.435s 2022-11-23T02:22:33.4522793Z 2022-11-23T02:22:33.4522872Z OK 2022-11-23T02:22:33.4522878Z 2022-11-23T02:22:33.4522998Z Generating XML reports... 2022-11-23T02:22:33.4523493Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021421.xml 2022-11-23T02:22:33.4523828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4524231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4524400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4524808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4525001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4525011Z 2022-11-23T02:22:33.4525112Z Running tests... 2022-11-23T02:22:33.4525395Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4526413Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.518s) 2022-11-23T02:22:33.4526422Z 2022-11-23T02:22:33.4526704Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4526809Z Ran 1 test in 0.518s 2022-11-23T02:22:33.4526816Z 2022-11-23T02:22:33.4526916Z OK (skipped=1) 2022-11-23T02:22:33.4526922Z 2022-11-23T02:22:33.4527040Z Generating XML reports... 2022-11-23T02:22:33.4527539Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021428.xml 2022-11-23T02:22:33.4527937Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4528342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4528523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4528937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4529113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4529134Z 2022-11-23T02:22:33.4529221Z Running tests... 2022-11-23T02:22:33.4529509Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4530011Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:22:33.4530020Z 2022-11-23T02:22:33.4530302Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4530404Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4530411Z 2022-11-23T02:22:33.4530507Z OK (skipped=1) 2022-11-23T02:22:33.4530513Z 2022-11-23T02:22:33.4530629Z Generating XML reports... 2022-11-23T02:22:33.4531159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021432.xml 2022-11-23T02:22:33.4531501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4531897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4532069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4532477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4532674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4532681Z 2022-11-23T02:22:33.4532783Z Running tests... 2022-11-23T02:22:33.4533071Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4534086Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.532s) 2022-11-23T02:22:33.4534094Z 2022-11-23T02:22:33.4534378Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4534481Z Ran 1 test in 0.532s 2022-11-23T02:22:33.4534488Z 2022-11-23T02:22:33.4534587Z OK (skipped=1) 2022-11-23T02:22:33.4534593Z 2022-11-23T02:22:33.4534709Z Generating XML reports... 2022-11-23T02:22:33.4535204Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021435.xml 2022-11-23T02:22:33.4535545Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4535945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4536122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4536525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4536700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4536717Z 2022-11-23T02:22:33.4536803Z Running tests... 2022-11-23T02:22:33.4537084Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4537423Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49950 2022-11-23T02:22:33.4537722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49951 2022-11-23T02:22:33.4537998Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4538405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4538574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4538983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4539170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4539426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4539824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4540004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4540410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4540648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4540901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4541341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4541773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4542011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4542247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4542511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddh8cqc6 2022-11-23T02:22:33.4542790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddh8cqc6/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4543052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwzu9qmm 2022-11-23T02:22:33.4543313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwzu9qmm/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4543685Z STAGE:2022-11-23 02:14:41 49951:49951 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4544041Z STAGE:2022-11-23 02:14:41 49950:49950 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4544731Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.4544839Z warnings.warn( 2022-11-23T02:22:33.4545084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4545770Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:22:33.4545868Z warnings.warn( 2022-11-23T02:22:33.4546104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:22:33.4546471Z STAGE:2022-11-23 02:14:41 49951:49951 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4546828Z STAGE:2022-11-23 02:14:41 49950:49950 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4547208Z STAGE:2022-11-23 02:14:41 49951:49951 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4547637Z STAGE:2022-11-23 02:14:41 49950:49950 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4547883Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:22:33.4548132Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:22:33.4548492Z STAGE:2022-11-23 02:14:41 49950:49950 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4548851Z STAGE:2022-11-23 02:14:41 49950:49950 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4549228Z STAGE:2022-11-23 02:14:41 49950:49950 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4549323Z ok (4.555s) 2022-11-23T02:22:33.4549330Z 2022-11-23T02:22:33.4549619Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4549727Z Ran 1 test in 4.556s 2022-11-23T02:22:33.4549734Z 2022-11-23T02:22:33.4549818Z OK 2022-11-23T02:22:33.4549824Z 2022-11-23T02:22:33.4549937Z Generating XML reports... 2022-11-23T02:22:33.4550413Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021438.xml 2022-11-23T02:22:33.4550798Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4551201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4551377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4551786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4551978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4551985Z 2022-11-23T02:22:33.4552084Z Running tests... 2022-11-23T02:22:33.4552368Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4552708Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50150 2022-11-23T02:22:33.4552936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50151 2022-11-23T02:22:33.4553214Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4553613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4553790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4554203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4554394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4554650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4555055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4555231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4555644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4555837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4556091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4556525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4556945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4557231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4557449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4557716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd66xiiqy 2022-11-23T02:22:33.4557995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd66xiiqy/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4558252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9yyndpd 2022-11-23T02:22:33.4558529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9yyndpd/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4558622Z ok (4.556s) 2022-11-23T02:22:33.4558629Z 2022-11-23T02:22:33.4558919Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4559024Z Ran 1 test in 4.556s 2022-11-23T02:22:33.4559034Z 2022-11-23T02:22:33.4559114Z OK 2022-11-23T02:22:33.4559120Z 2022-11-23T02:22:33.4559239Z Generating XML reports... 2022-11-23T02:22:33.4559734Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021446.xml 2022-11-23T02:22:33.4560123Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4560610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4560788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4561202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4561394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4561402Z 2022-11-23T02:22:33.4561505Z Running tests... 2022-11-23T02:22:33.4561790Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4562139Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50340 2022-11-23T02:22:33.4562362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50341 2022-11-23T02:22:33.4562641Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4563038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4563198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4563607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4563800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4564047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4564449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4564622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4565027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4565215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4565462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4565894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4566326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4566666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4566900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4567166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8dtkh4e 2022-11-23T02:22:33.4567450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8dtkh4e/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4567714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpughz93rm 2022-11-23T02:22:33.4567989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpughz93rm/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4568087Z ok (4.252s) 2022-11-23T02:22:33.4568093Z 2022-11-23T02:22:33.4568385Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4568495Z Ran 1 test in 4.252s 2022-11-23T02:22:33.4568502Z 2022-11-23T02:22:33.4568584Z OK 2022-11-23T02:22:33.4568593Z 2022-11-23T02:22:33.4568710Z Generating XML reports... 2022-11-23T02:22:33.4569192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021453.xml 2022-11-23T02:22:33.4569531Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4569990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4570167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4570581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4570774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4570781Z 2022-11-23T02:22:33.4570879Z Running tests... 2022-11-23T02:22:33.4571165Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4572185Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.516s) 2022-11-23T02:22:33.4572194Z 2022-11-23T02:22:33.4572474Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4572578Z Ran 1 test in 0.517s 2022-11-23T02:22:33.4572585Z 2022-11-23T02:22:33.4572684Z OK (skipped=1) 2022-11-23T02:22:33.4572690Z 2022-11-23T02:22:33.4572810Z Generating XML reports... 2022-11-23T02:22:33.4573303Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021501.xml 2022-11-23T02:22:33.4573639Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4574048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4574220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4574638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4574831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4574839Z 2022-11-23T02:22:33.4574937Z Running tests... 2022-11-23T02:22:33.4575220Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4576196Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.533s) 2022-11-23T02:22:33.4576251Z 2022-11-23T02:22:33.4576533Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4576632Z Ran 1 test in 0.533s 2022-11-23T02:22:33.4576639Z 2022-11-23T02:22:33.4576734Z OK (skipped=1) 2022-11-23T02:22:33.4576740Z 2022-11-23T02:22:33.4576856Z Generating XML reports... 2022-11-23T02:22:33.4577340Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021504.xml 2022-11-23T02:22:33.4577675Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4578076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4578247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4578659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4578855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4578861Z 2022-11-23T02:22:33.4578958Z Running tests... 2022-11-23T02:22:33.4579240Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4580319Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.517s) 2022-11-23T02:22:33.4580328Z 2022-11-23T02:22:33.4580611Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4580718Z Ran 1 test in 0.517s 2022-11-23T02:22:33.4580725Z 2022-11-23T02:22:33.4580822Z OK (skipped=1) 2022-11-23T02:22:33.4580828Z 2022-11-23T02:22:33.4580944Z Generating XML reports... 2022-11-23T02:22:33.4581435Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021508.xml 2022-11-23T02:22:33.4581768Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4582166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4582338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4582745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4582932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4582939Z 2022-11-23T02:22:33.4583037Z Running tests... 2022-11-23T02:22:33.4583318Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4583846Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:22:33.4583856Z 2022-11-23T02:22:33.4584137Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4584240Z Ran 1 test in 0.003s 2022-11-23T02:22:33.4584247Z 2022-11-23T02:22:33.4584350Z OK (skipped=1) 2022-11-23T02:22:33.4584356Z 2022-11-23T02:22:33.4584474Z Generating XML reports... 2022-11-23T02:22:33.4584952Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021511.xml 2022-11-23T02:22:33.4585293Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4585689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4585865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4586277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4586516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4586522Z 2022-11-23T02:22:33.4586624Z Running tests... 2022-11-23T02:22:33.4586911Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4587253Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50758 2022-11-23T02:22:33.4587472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50759 2022-11-23T02:22:33.4587745Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4588139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4588314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4588722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4588910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4589201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4589610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4589783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4590188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4590380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4590636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4591080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4591515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4591736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4591975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4592387Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:22:33.4592668Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:22:33.4592930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6_7w45j 2022-11-23T02:22:33.4593204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6_7w45j/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4593614Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:22:33.4593885Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:22:33.4594146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx70n9rj1 2022-11-23T02:22:33.4594422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx70n9rj1/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4594517Z ok (4.351s) 2022-11-23T02:22:33.4594524Z 2022-11-23T02:22:33.4594811Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4594912Z Ran 1 test in 4.352s 2022-11-23T02:22:33.4594919Z 2022-11-23T02:22:33.4595002Z OK 2022-11-23T02:22:33.4595010Z 2022-11-23T02:22:33.4595125Z Generating XML reports... 2022-11-23T02:22:33.4595612Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021515.xml 2022-11-23T02:22:33.4596007Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4596402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4596575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4596984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4597175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4597182Z 2022-11-23T02:22:33.4597282Z Running tests... 2022-11-23T02:22:33.4597558Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4597893Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50944 2022-11-23T02:22:33.4598116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50945 2022-11-23T02:22:33.4598399Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4598796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4599017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4599434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4599628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4599881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4600358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4600533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4600945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4601133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4601386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4601817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4602249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4602478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4602722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.4602947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4603186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.4603613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4604042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4604121Z ok (3.751s) 2022-11-23T02:22:33.4604146Z 2022-11-23T02:22:33.4604422Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4604531Z Ran 1 test in 3.751s 2022-11-23T02:22:33.4604538Z 2022-11-23T02:22:33.4604620Z OK 2022-11-23T02:22:33.4604627Z 2022-11-23T02:22:33.4604746Z Generating XML reports... 2022-11-23T02:22:33.4605235Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021522.xml 2022-11-23T02:22:33.4605640Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4606039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4606216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4606624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4606815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4606822Z 2022-11-23T02:22:33.4606921Z Running tests... 2022-11-23T02:22:33.4607208Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4607535Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51125 2022-11-23T02:22:33.4607758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51126 2022-11-23T02:22:33.4608041Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4608444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4608695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4609109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4609303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4609548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4609943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4610112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4610510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4610699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4610952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4611381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4611810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4612049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4612291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.4612518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4612769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.4613199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4613627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4613724Z ok (3.861s) 2022-11-23T02:22:33.4613731Z 2022-11-23T02:22:33.4614019Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4614125Z Ran 1 test in 3.861s 2022-11-23T02:22:33.4614132Z 2022-11-23T02:22:33.4614218Z OK 2022-11-23T02:22:33.4614224Z 2022-11-23T02:22:33.4614342Z Generating XML reports... 2022-11-23T02:22:33.4614834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021529.xml 2022-11-23T02:22:33.4615232Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4615635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4615806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4616212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4616398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4616405Z 2022-11-23T02:22:33.4616508Z Running tests... 2022-11-23T02:22:33.4616777Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4617796Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.513s) 2022-11-23T02:22:33.4617823Z 2022-11-23T02:22:33.4618103Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4618195Z Ran 1 test in 0.514s 2022-11-23T02:22:33.4618261Z 2022-11-23T02:22:33.4618349Z OK (skipped=1) 2022-11-23T02:22:33.4618355Z 2022-11-23T02:22:33.4618473Z Generating XML reports... 2022-11-23T02:22:33.4618960Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021536.xml 2022-11-23T02:22:33.4619299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4619694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4619868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4620278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4620467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4620474Z 2022-11-23T02:22:33.4620569Z Running tests... 2022-11-23T02:22:33.4620850Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4621340Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:22:33.4621348Z 2022-11-23T02:22:33.4621623Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4621722Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4621729Z 2022-11-23T02:22:33.4621825Z OK (skipped=1) 2022-11-23T02:22:33.4621832Z 2022-11-23T02:22:33.4621947Z Generating XML reports... 2022-11-23T02:22:33.4622435Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021539.xml 2022-11-23T02:22:33.4622771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4623167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4623340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4623746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4623936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4623943Z 2022-11-23T02:22:33.4624040Z Running tests... 2022-11-23T02:22:33.4624308Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4624659Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51422 2022-11-23T02:22:33.4624928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51423 2022-11-23T02:22:33.4625207Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4625614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4625791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4626203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4626394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4626645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4627040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4627215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4627624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4627814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4628108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4628554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4628977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4629209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4629442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4629533Z ok (3.850s) 2022-11-23T02:22:33.4629540Z 2022-11-23T02:22:33.4629824Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4629928Z Ran 1 test in 3.850s 2022-11-23T02:22:33.4629935Z 2022-11-23T02:22:33.4630014Z OK 2022-11-23T02:22:33.4630020Z 2022-11-23T02:22:33.4630126Z Generating XML reports... 2022-11-23T02:22:33.4630623Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021543.xml 2022-11-23T02:22:33.4630960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4631363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4631534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4631941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4632132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4632139Z 2022-11-23T02:22:33.4632232Z Running tests... 2022-11-23T02:22:33.4632509Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4632763Z test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.4632771Z 2022-11-23T02:22:33.4633045Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4633152Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4633159Z 2022-11-23T02:22:33.4633260Z OK (skipped=1) 2022-11-23T02:22:33.4633266Z 2022-11-23T02:22:33.4633382Z Generating XML reports... 2022-11-23T02:22:33.4633879Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021549.xml 2022-11-23T02:22:33.4634213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4634657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4634830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4635244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4635432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4635438Z 2022-11-23T02:22:33.4635537Z Running tests... 2022-11-23T02:22:33.4635816Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4636072Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4636090Z 2022-11-23T02:22:33.4636355Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4636454Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4636463Z 2022-11-23T02:22:33.4636555Z OK (skipped=1) 2022-11-23T02:22:33.4636561Z 2022-11-23T02:22:33.4636674Z Generating XML reports... 2022-11-23T02:22:33.4637154Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021553.xml 2022-11-23T02:22:33.4637535Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4637932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4638102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4638502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4638693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4638700Z 2022-11-23T02:22:33.4638800Z Running tests... 2022-11-23T02:22:33.4639082Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4639407Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51717 2022-11-23T02:22:33.4639627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51718 2022-11-23T02:22:33.4639908Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4640379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4640557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4640964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4641158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4641404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4641804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4641979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4642379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4642568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4642818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4643253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4643682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4643972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4644205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4644571Z STAGE:2022-11-23 02:15:58 51718:51718 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4644926Z STAGE:2022-11-23 02:15:58 51717:51717 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4645291Z STAGE:2022-11-23 02:15:59 51717:51717 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4645671Z STAGE:2022-11-23 02:15:59 51717:51717 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4645914Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.4646283Z STAGE:2022-11-23 02:15:59 51718:51718 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4646663Z STAGE:2022-11-23 02:15:59 51718:51718 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4646906Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.4647262Z STAGE:2022-11-23 02:15:59 51717:51717 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4647661Z STAGE:2022-11-23 02:15:59 51718:51718 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4648030Z STAGE:2022-11-23 02:15:59 51718:51718 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4648406Z STAGE:2022-11-23 02:15:59 51718:51718 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4648764Z STAGE:2022-11-23 02:15:59 51717:51717 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4649245Z STAGE:2022-11-23 02:15:59 51717:51717 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4649317Z ok (4.484s) 2022-11-23T02:22:33.4649323Z 2022-11-23T02:22:33.4649528Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4649593Z Ran 1 test in 4.484s 2022-11-23T02:22:33.4649611Z 2022-11-23T02:22:33.4649660Z OK 2022-11-23T02:22:33.4649664Z 2022-11-23T02:22:33.4649747Z Generating XML reports... 2022-11-23T02:22:33.4650097Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021556.xml 2022-11-23T02:22:33.4650340Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4650620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4650741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4651032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4651171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4651176Z 2022-11-23T02:22:33.4651248Z Running tests... 2022-11-23T02:22:33.4651451Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4651644Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.4651649Z 2022-11-23T02:22:33.4651850Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4651926Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4651931Z 2022-11-23T02:22:33.4652005Z OK (skipped=1) 2022-11-23T02:22:33.4652010Z 2022-11-23T02:22:33.4652092Z Generating XML reports... 2022-11-23T02:22:33.4652436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021603.xml 2022-11-23T02:22:33.4652673Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4653004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4653126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4653417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4653548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4653553Z 2022-11-23T02:22:33.4653620Z Running tests... 2022-11-23T02:22:33.4653812Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4654000Z test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.4654005Z 2022-11-23T02:22:33.4654203Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4654280Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4654285Z 2022-11-23T02:22:33.4654360Z OK (skipped=1) 2022-11-23T02:22:33.4654365Z 2022-11-23T02:22:33.4654445Z Generating XML reports... 2022-11-23T02:22:33.4654789Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021606.xml 2022-11-23T02:22:33.4655062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4655346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4655468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4655758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4655895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4655899Z 2022-11-23T02:22:33.4655973Z Running tests... 2022-11-23T02:22:33.4656172Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4656404Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52025 2022-11-23T02:22:33.4656564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52026 2022-11-23T02:22:33.4656764Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4657046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4657170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4657462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4657599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4657775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4658043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4658164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4658454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4658589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4658765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4659078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4659381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4659552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4659758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4659827Z ok (4.650s) 2022-11-23T02:22:33.4659831Z 2022-11-23T02:22:33.4660042Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4660119Z Ran 1 test in 4.651s 2022-11-23T02:22:33.4660124Z 2022-11-23T02:22:33.4660186Z OK 2022-11-23T02:22:33.4660191Z 2022-11-23T02:22:33.4660273Z Generating XML reports... 2022-11-23T02:22:33.4660623Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021609.xml 2022-11-23T02:22:33.4660863Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4661153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4661285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4661578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4661721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4661725Z 2022-11-23T02:22:33.4661838Z Running tests... 2022-11-23T02:22:33.4662044Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4662750Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.559s) 2022-11-23T02:22:33.4662756Z 2022-11-23T02:22:33.4662962Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4663045Z Ran 1 test in 0.559s 2022-11-23T02:22:33.4663049Z 2022-11-23T02:22:33.4663113Z OK (skipped=1) 2022-11-23T02:22:33.4663131Z 2022-11-23T02:22:33.4663205Z Generating XML reports... 2022-11-23T02:22:33.4663561Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021617.xml 2022-11-23T02:22:33.4663806Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4664088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4664212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4664504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4664641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4664645Z 2022-11-23T02:22:33.4664722Z Running tests... 2022-11-23T02:22:33.4664926Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4665155Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52269 2022-11-23T02:22:33.4665316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52270 2022-11-23T02:22:33.4665511Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4665794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4665920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4666212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4666352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4666571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4666855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4666980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4667271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4667411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4667595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4667893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4668204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4668380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4668555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.4668752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4668924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.4669225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4669526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4669593Z ok (3.841s) 2022-11-23T02:22:33.4669598Z 2022-11-23T02:22:33.4669800Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4669879Z Ran 1 test in 3.841s 2022-11-23T02:22:33.4669884Z 2022-11-23T02:22:33.4669944Z OK 2022-11-23T02:22:33.4669948Z 2022-11-23T02:22:33.4670030Z Generating XML reports... 2022-11-23T02:22:33.4670382Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021620.xml 2022-11-23T02:22:33.4670625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4670908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4671036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4671328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4671471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4671476Z 2022-11-23T02:22:33.4671551Z Running tests... 2022-11-23T02:22:33.4671758Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4671988Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52450 2022-11-23T02:22:33.4672146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52451 2022-11-23T02:22:33.4672331Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4672612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4672737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4673027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4673163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4673390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4673672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4673795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4674085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4674220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4674399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4674706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4675009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4675181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4675519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4675590Z ok (4.344s) 2022-11-23T02:22:33.4675595Z 2022-11-23T02:22:33.4675858Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4675941Z Ran 1 test in 4.345s 2022-11-23T02:22:33.4675946Z 2022-11-23T02:22:33.4676015Z OK 2022-11-23T02:22:33.4676019Z 2022-11-23T02:22:33.4676111Z Generating XML reports... 2022-11-23T02:22:33.4676481Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021627.xml 2022-11-23T02:22:33.4676738Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4677024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4677162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4677470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4677613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4677620Z 2022-11-23T02:22:33.4677696Z Running tests... 2022-11-23T02:22:33.4677910Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4678144Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52636 2022-11-23T02:22:33.4678309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52637 2022-11-23T02:22:33.4678514Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4678809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4678943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4679246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4679385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4679573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4679870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4680002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4680393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4680707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4680954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4681291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4681628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4681807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4681989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4682049Z ok (3.941s) 2022-11-23T02:22:33.4682055Z 2022-11-23T02:22:33.4682274Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4682356Z Ran 1 test in 3.941s 2022-11-23T02:22:33.4682361Z 2022-11-23T02:22:33.4682426Z OK 2022-11-23T02:22:33.4682430Z 2022-11-23T02:22:33.4682526Z Generating XML reports... 2022-11-23T02:22:33.4682908Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021635.xml 2022-11-23T02:22:33.4683170Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4683520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4683656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4683973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4684116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4684122Z 2022-11-23T02:22:33.4684200Z Running tests... 2022-11-23T02:22:33.4684423Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4684684Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52815 2022-11-23T02:22:33.4684860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52816 2022-11-23T02:22:33.4685077Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4685381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4685515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4685825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4685971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4686159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4686464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4686586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4686901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4687051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4687246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4687576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4687902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4688081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4688310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.4688483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4688672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.4688996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4689313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4689391Z ok (3.955s) 2022-11-23T02:22:33.4689396Z 2022-11-23T02:22:33.4689618Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4689701Z Ran 1 test in 3.955s 2022-11-23T02:22:33.4689707Z 2022-11-23T02:22:33.4689768Z OK 2022-11-23T02:22:33.4689772Z 2022-11-23T02:22:33.4689868Z Generating XML reports... 2022-11-23T02:22:33.4690246Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021642.xml 2022-11-23T02:22:33.4690514Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4690963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4691099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4691408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4691541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4691559Z 2022-11-23T02:22:33.4691623Z Running tests... 2022-11-23T02:22:33.4691842Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4692096Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52996 2022-11-23T02:22:33.4692269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52997 2022-11-23T02:22:33.4692482Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4692887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4693017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4693307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4693445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4693624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4693908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4694030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4694322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4694454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4694632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4694940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4695242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4695412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4695624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:22:33.4695786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4695955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:22:33.4696260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4696562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:22:33.4696618Z ok (3.640s) 2022-11-23T02:22:33.4696633Z 2022-11-23T02:22:33.4697001Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4697083Z Ran 1 test in 3.640s 2022-11-23T02:22:33.4697088Z 2022-11-23T02:22:33.4697153Z OK 2022-11-23T02:22:33.4697158Z 2022-11-23T02:22:33.4697267Z Generating XML reports... 2022-11-23T02:22:33.4697640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021649.xml 2022-11-23T02:22:33.4697893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4698235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4698370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4698681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4698826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4698834Z 2022-11-23T02:22:33.4698911Z Running tests... 2022-11-23T02:22:33.4699124Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4699474Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:22:33.4699482Z 2022-11-23T02:22:33.4699693Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4699772Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4699778Z 2022-11-23T02:22:33.4699852Z OK (skipped=1) 2022-11-23T02:22:33.4699857Z 2022-11-23T02:22:33.4699946Z Generating XML reports... 2022-11-23T02:22:33.4700314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021655.xml 2022-11-23T02:22:33.4700565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4700860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4700990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4701286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4701432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4701449Z 2022-11-23T02:22:33.4701513Z Running tests... 2022-11-23T02:22:33.4701729Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4701914Z test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s) 2022-11-23T02:22:33.4701918Z 2022-11-23T02:22:33.4702127Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4702206Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4702211Z 2022-11-23T02:22:33.4702286Z OK (skipped=1) 2022-11-23T02:22:33.4702291Z 2022-11-23T02:22:33.4702377Z Generating XML reports... 2022-11-23T02:22:33.4702740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021658.xml 2022-11-23T02:22:33.4703038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4703340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4703474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4703779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4703920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4703925Z 2022-11-23T02:22:33.4703999Z Running tests... 2022-11-23T02:22:33.4704205Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4704386Z test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:22:33.4704391Z 2022-11-23T02:22:33.4704592Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4704675Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4704680Z 2022-11-23T02:22:33.4704754Z OK (skipped=1) 2022-11-23T02:22:33.4704759Z 2022-11-23T02:22:33.4704848Z Generating XML reports... 2022-11-23T02:22:33.4705197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021701.xml 2022-11-23T02:22:33.4705487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4705791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4705923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4706226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4706375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4706379Z 2022-11-23T02:22:33.4706461Z Running tests... 2022-11-23T02:22:33.4706682Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4706890Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:22:33.4706895Z 2022-11-23T02:22:33.4725426Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4725569Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4725576Z 2022-11-23T02:22:33.4725663Z OK (skipped=1) 2022-11-23T02:22:33.4725668Z 2022-11-23T02:22:33.4725767Z Generating XML reports... 2022-11-23T02:22:33.4726231Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021704.xml 2022-11-23T02:22:33.4726493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4726811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4726963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4727289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4727451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4727456Z 2022-11-23T02:22:33.4727547Z Running tests... 2022-11-23T02:22:33.4727773Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4727984Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:22:33.4727990Z 2022-11-23T02:22:33.4728205Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4728285Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4728290Z 2022-11-23T02:22:33.4728370Z OK (skipped=1) 2022-11-23T02:22:33.4728374Z 2022-11-23T02:22:33.4728464Z Generating XML reports... 2022-11-23T02:22:33.4728828Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021707.xml 2022-11-23T02:22:33.4729240Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4729542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4729678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4729990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4730138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4730143Z 2022-11-23T02:22:33.4730224Z Running tests... 2022-11-23T02:22:33.4730445Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4730823Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:22:33.4730832Z 2022-11-23T02:22:33.4731045Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4731132Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4731138Z 2022-11-23T02:22:33.4731223Z OK (skipped=1) 2022-11-23T02:22:33.4731227Z 2022-11-23T02:22:33.4731383Z Generating XML reports... 2022-11-23T02:22:33.4731757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021710.xml 2022-11-23T02:22:33.4732011Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4732317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4732454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4732760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4732910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4732915Z 2022-11-23T02:22:33.4732991Z Running tests... 2022-11-23T02:22:33.4733207Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4733609Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:22:33.4733616Z 2022-11-23T02:22:33.4733832Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4733902Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4733922Z 2022-11-23T02:22:33.4733987Z OK (skipped=1) 2022-11-23T02:22:33.4734002Z 2022-11-23T02:22:33.4734249Z Generating XML reports... 2022-11-23T02:22:33.4734621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021713.xml 2022-11-23T02:22:33.4734876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4735181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4735318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4735631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4735781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4735786Z 2022-11-23T02:22:33.4735867Z Running tests... 2022-11-23T02:22:33.4736086Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4736429Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:22:33.4736434Z 2022-11-23T02:22:33.4736651Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4736776Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4736781Z 2022-11-23T02:22:33.4736860Z OK (skipped=1) 2022-11-23T02:22:33.4736864Z 2022-11-23T02:22:33.4736958Z Generating XML reports... 2022-11-23T02:22:33.4737333Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021715.xml 2022-11-23T02:22:33.4737592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4737917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4738053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4738360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4738504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4738512Z 2022-11-23T02:22:33.4738593Z Running tests... 2022-11-23T02:22:33.4738797Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4739117Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:22:33.4739134Z 2022-11-23T02:22:33.4739398Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4739478Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4739482Z 2022-11-23T02:22:33.4739559Z OK (skipped=1) 2022-11-23T02:22:33.4739564Z 2022-11-23T02:22:33.4739658Z Generating XML reports... 2022-11-23T02:22:33.4740025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021718.xml 2022-11-23T02:22:33.4740280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4740582Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4740720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4741028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4741180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4741185Z 2022-11-23T02:22:33.4741267Z Running tests... 2022-11-23T02:22:33.4741487Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4741838Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:22:33.4741843Z 2022-11-23T02:22:33.4742057Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4742143Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4742148Z 2022-11-23T02:22:33.4742229Z OK (skipped=1) 2022-11-23T02:22:33.4742236Z 2022-11-23T02:22:33.4742332Z Generating XML reports... 2022-11-23T02:22:33.4742704Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021721.xml 2022-11-23T02:22:33.4742963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4743268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4743405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4743703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4743852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4743870Z 2022-11-23T02:22:33.4743937Z Running tests... 2022-11-23T02:22:33.4744154Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4744539Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:22:33.4744544Z 2022-11-23T02:22:33.4744810Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4744892Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4744897Z 2022-11-23T02:22:33.4744978Z OK (skipped=1) 2022-11-23T02:22:33.4744983Z 2022-11-23T02:22:33.4745077Z Generating XML reports... 2022-11-23T02:22:33.4745447Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021724.xml 2022-11-23T02:22:33.4745700Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4746000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4746131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4746442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4746589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4746594Z 2022-11-23T02:22:33.4746676Z Running tests... 2022-11-23T02:22:33.4746956Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4747299Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:22:33.4747305Z 2022-11-23T02:22:33.4747521Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4747604Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4747609Z 2022-11-23T02:22:33.4747691Z OK (skipped=1) 2022-11-23T02:22:33.4747695Z 2022-11-23T02:22:33.4747787Z Generating XML reports... 2022-11-23T02:22:33.4748156Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021727.xml 2022-11-23T02:22:33.4748415Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4748703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4748841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4749155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4749300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4749304Z 2022-11-23T02:22:33.4749383Z Running tests... 2022-11-23T02:22:33.4749601Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4749859Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53873 2022-11-23T02:22:33.4750038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53874 2022-11-23T02:22:33.4750250Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4750552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4750687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4750998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4751141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4751332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4751637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4751810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4752127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4752273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4752470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4752802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4753130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4753312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4753492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4753557Z ok (4.254s) 2022-11-23T02:22:33.4753563Z 2022-11-23T02:22:33.4753783Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4753867Z Ran 1 test in 4.255s 2022-11-23T02:22:33.4753872Z 2022-11-23T02:22:33.4753939Z OK 2022-11-23T02:22:33.4753943Z 2022-11-23T02:22:33.4754038Z Generating XML reports... 2022-11-23T02:22:33.4754450Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021730.xml 2022-11-23T02:22:33.4754711Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4755016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4755151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4755460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4755611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4755616Z 2022-11-23T02:22:33.4755695Z Running tests... 2022-11-23T02:22:33.4777160Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4777545Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54063 2022-11-23T02:22:33.4777729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54064 2022-11-23T02:22:33.4777948Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4778367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4778509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4778850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4779008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4779204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4779520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4779642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4779961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4780117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4780312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4780650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4781177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4781362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4781548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4781628Z ok (4.540s) 2022-11-23T02:22:33.4781637Z 2022-11-23T02:22:33.4781861Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4781947Z Ran 1 test in 4.541s 2022-11-23T02:22:33.4781953Z 2022-11-23T02:22:33.4782018Z OK 2022-11-23T02:22:33.4782022Z 2022-11-23T02:22:33.4782119Z Generating XML reports... 2022-11-23T02:22:33.4782499Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021737.xml 2022-11-23T02:22:33.4782759Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4783068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4783205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4783627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4783784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4783790Z 2022-11-23T02:22:33.4783873Z Running tests... 2022-11-23T02:22:33.4784115Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4784426Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54253 2022-11-23T02:22:33.4784593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54254 2022-11-23T02:22:33.4784837Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4805008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4805185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4805598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4805748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4805945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4806264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4806405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4806730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4806877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4807074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4807413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4807746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4807928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4808107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4808181Z ok (4.229s) 2022-11-23T02:22:33.4808190Z 2022-11-23T02:22:33.4808415Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4808677Z Ran 1 test in 4.229s 2022-11-23T02:22:33.4808683Z 2022-11-23T02:22:33.4808750Z OK 2022-11-23T02:22:33.4808755Z 2022-11-23T02:22:33.4808851Z Generating XML reports... 2022-11-23T02:22:33.4809250Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021745.xml 2022-11-23T02:22:33.4809508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4809807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4809937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4810248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4810391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4810400Z 2022-11-23T02:22:33.4810472Z Running tests... 2022-11-23T02:22:33.4810688Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4811049Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54443 2022-11-23T02:22:33.4811276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54444 2022-11-23T02:22:33.4811480Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4811776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4811905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4812203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4812342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4812526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4812817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4812945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4813245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4813381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4813561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4813881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4814197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4814370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4814536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4814596Z ok (4.242s) 2022-11-23T02:22:33.4814604Z 2022-11-23T02:22:33.4814815Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4814889Z Ran 1 test in 4.242s 2022-11-23T02:22:33.4814895Z 2022-11-23T02:22:33.4814953Z OK 2022-11-23T02:22:33.4814957Z 2022-11-23T02:22:33.4815041Z Generating XML reports... 2022-11-23T02:22:33.4815405Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021752.xml 2022-11-23T02:22:33.4815649Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4815945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4816239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4816545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4816687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4816692Z 2022-11-23T02:22:33.4816763Z Running tests... 2022-11-23T02:22:33.4816970Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4817719Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.512s) 2022-11-23T02:22:33.4817727Z 2022-11-23T02:22:33.4817933Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4818005Z Ran 1 test in 0.512s 2022-11-23T02:22:33.4818010Z 2022-11-23T02:22:33.4818078Z OK (skipped=1) 2022-11-23T02:22:33.4818083Z 2022-11-23T02:22:33.4818166Z Generating XML reports... 2022-11-23T02:22:33.4818570Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021759.xml 2022-11-23T02:22:33.4818821Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4819130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4819258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4819564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4819703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4819709Z 2022-11-23T02:22:33.4819779Z Running tests... 2022-11-23T02:22:33.4820144Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4820342Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:22:33.4820347Z 2022-11-23T02:22:33.4820559Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4820633Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4820638Z 2022-11-23T02:22:33.4820708Z OK (skipped=1) 2022-11-23T02:22:33.4820712Z 2022-11-23T02:22:33.4820797Z Generating XML reports... 2022-11-23T02:22:33.4821167Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021803.xml 2022-11-23T02:22:33.4821422Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4821723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4821851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4822158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4822303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4822309Z 2022-11-23T02:22:33.4822380Z Running tests... 2022-11-23T02:22:33.4822593Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4822806Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:22:33.4822812Z 2022-11-23T02:22:33.4823020Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4823094Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4823099Z 2022-11-23T02:22:33.4823210Z OK (skipped=1) 2022-11-23T02:22:33.4823214Z 2022-11-23T02:22:33.4823299Z Generating XML reports... 2022-11-23T02:22:33.4823674Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021806.xml 2022-11-23T02:22:33.4823930Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4824230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4824352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4824660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4824800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4824805Z 2022-11-23T02:22:33.4824876Z Running tests... 2022-11-23T02:22:33.4825088Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4825335Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:22:33.4825340Z 2022-11-23T02:22:33.4825549Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4825658Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4825664Z 2022-11-23T02:22:33.4825734Z OK (skipped=1) 2022-11-23T02:22:33.4825739Z 2022-11-23T02:22:33.4825822Z Generating XML reports... 2022-11-23T02:22:33.4826197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021808.xml 2022-11-23T02:22:33.4826449Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4826749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4826877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4827192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4827332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4827337Z 2022-11-23T02:22:33.4827408Z Running tests... 2022-11-23T02:22:33.4827625Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4827907Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54863 2022-11-23T02:22:33.4828072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54864 2022-11-23T02:22:33.4848908Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4849240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4849374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4849684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4849822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4850010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4850312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4850440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4850748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4850889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4851143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4851580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4851900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4852069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4852237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4852302Z ok (3.744s) 2022-11-23T02:22:33.4852308Z 2022-11-23T02:22:33.4852518Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4852591Z Ran 1 test in 3.745s 2022-11-23T02:22:33.4852596Z 2022-11-23T02:22:33.4852654Z OK 2022-11-23T02:22:33.4852659Z 2022-11-23T02:22:33.4852741Z Generating XML reports... 2022-11-23T02:22:33.4853107Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021811.xml 2022-11-23T02:22:33.4853355Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4853691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4853974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4854274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4854404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4854414Z 2022-11-23T02:22:33.4854479Z Running tests... 2022-11-23T02:22:33.4854683Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4854952Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55042 2022-11-23T02:22:33.4855116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55043 2022-11-23T02:22:33.4855321Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4855610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4855734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4856032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4856168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4856350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4856642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4856765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4857063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4857200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4857378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4857696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4858011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4858180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4858410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4858475Z ok (3.856s) 2022-11-23T02:22:33.4858481Z 2022-11-23T02:22:33.4858692Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4858792Z Ran 1 test in 3.856s 2022-11-23T02:22:33.4858798Z 2022-11-23T02:22:33.4858854Z OK 2022-11-23T02:22:33.4858864Z 2022-11-23T02:22:33.4858941Z Generating XML reports... 2022-11-23T02:22:33.4859310Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021818.xml 2022-11-23T02:22:33.4859557Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4859848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4859973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4860273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4860413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4860418Z 2022-11-23T02:22:33.4860493Z Running tests... 2022-11-23T02:22:33.4860744Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4860954Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:22:33.4860959Z 2022-11-23T02:22:33.4861168Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4861242Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4861247Z 2022-11-23T02:22:33.4861320Z OK (skipped=1) 2022-11-23T02:22:33.4861324Z 2022-11-23T02:22:33.4861408Z Generating XML reports... 2022-11-23T02:22:33.4861774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021825.xml 2022-11-23T02:22:33.4862024Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4862317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4862442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4862745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4862884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4862889Z 2022-11-23T02:22:33.4862958Z Running tests... 2022-11-23T02:22:33.4863157Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4863385Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.001s) 2022-11-23T02:22:33.4863398Z 2022-11-23T02:22:33.4863598Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4863669Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4863674Z 2022-11-23T02:22:33.4863744Z OK (skipped=1) 2022-11-23T02:22:33.4863748Z 2022-11-23T02:22:33.4863833Z Generating XML reports... 2022-11-23T02:22:33.4864198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021828.xml 2022-11-23T02:22:33.4864447Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4864741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4864871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4865173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4865313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4865360Z 2022-11-23T02:22:33.4865434Z Running tests... 2022-11-23T02:22:33.4865643Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4866408Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:22:33.4866413Z 2022-11-23T02:22:33.4866623Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4866705Z Ran 1 test in 0.509s 2022-11-23T02:22:33.4866710Z 2022-11-23T02:22:33.4866791Z OK (skipped=1) 2022-11-23T02:22:33.4866795Z 2022-11-23T02:22:33.4866889Z Generating XML reports... 2022-11-23T02:22:33.4867257Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021831.xml 2022-11-23T02:22:33.4867516Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4867818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4867998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4868308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4868456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4868462Z 2022-11-23T02:22:33.4868539Z Running tests... 2022-11-23T02:22:33.4868740Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4869009Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55395 2022-11-23T02:22:33.4869181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55396 2022-11-23T02:22:33.4869395Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4869696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4869832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4870141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4870288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4870478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4870777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4870910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4871217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4871367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4871558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4871890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4872212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4872390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4872569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4872810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qe1tqwl 2022-11-23T02:22:33.4873020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qe1tqwl/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4873221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfsddhok 2022-11-23T02:22:33.4873429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfsddhok/_remote_module_non_scriptable.py 2022-11-23T02:22:33.4873506Z ok (6.545s) 2022-11-23T02:22:33.4873511Z 2022-11-23T02:22:33.4873734Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4873803Z Ran 1 test in 6.545s 2022-11-23T02:22:33.4873808Z 2022-11-23T02:22:33.4873875Z OK 2022-11-23T02:22:33.4873880Z 2022-11-23T02:22:33.4873970Z Generating XML reports... 2022-11-23T02:22:33.4874341Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021834.xml 2022-11-23T02:22:33.4874603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4874906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4875081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4875394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4875541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4875546Z 2022-11-23T02:22:33.4875625Z Running tests... 2022-11-23T02:22:33.4875842Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4876106Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55591 2022-11-23T02:22:33.4876275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55592 2022-11-23T02:22:33.4876487Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4876788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4876923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4877230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4877376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4877570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4877871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4878011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4878318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4878450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4878641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4878970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4879292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4900723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4901038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4901341Z ok (4.440s) 2022-11-23T02:22:33.4901350Z 2022-11-23T02:22:33.4901719Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4901825Z Ran 1 test in 4.441s 2022-11-23T02:22:33.4901832Z 2022-11-23T02:22:33.4901915Z OK 2022-11-23T02:22:33.4901921Z 2022-11-23T02:22:33.4902040Z Generating XML reports... 2022-11-23T02:22:33.4902529Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021844.xml 2022-11-23T02:22:33.4902847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4903233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4903400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4903788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4903977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4903984Z 2022-11-23T02:22:33.4904081Z Running tests... 2022-11-23T02:22:33.4904353Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4904768Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55781 2022-11-23T02:22:33.4904986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55782 2022-11-23T02:22:33.4905255Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4905623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4905792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4906177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4906366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4906607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4906985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4907153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4907538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4907716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4907952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4908362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4908769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4908991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4909213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4909303Z ok (4.343s) 2022-11-23T02:22:33.4909310Z 2022-11-23T02:22:33.4909581Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4909681Z Ran 1 test in 4.343s 2022-11-23T02:22:33.4909688Z 2022-11-23T02:22:33.4909768Z OK 2022-11-23T02:22:33.4909774Z 2022-11-23T02:22:33.4909885Z Generating XML reports... 2022-11-23T02:22:33.4910350Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021851.xml 2022-11-23T02:22:33.4910670Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4911157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4911313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4911711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4911891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4911898Z 2022-11-23T02:22:33.4911997Z Running tests... 2022-11-23T02:22:33.4912270Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4913227Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:22:33.4913238Z 2022-11-23T02:22:33.4913507Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4913607Z Ran 1 test in 0.509s 2022-11-23T02:22:33.4913613Z 2022-11-23T02:22:33.4913707Z OK (skipped=1) 2022-11-23T02:22:33.4913777Z 2022-11-23T02:22:33.4913894Z Generating XML reports... 2022-11-23T02:22:33.4914363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021859.xml 2022-11-23T02:22:33.4914681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4915057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4915223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4915608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4915791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4915798Z 2022-11-23T02:22:33.4915896Z Running tests... 2022-11-23T02:22:33.4916167Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4917140Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.523s) 2022-11-23T02:22:33.4917147Z 2022-11-23T02:22:33.4917414Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4917513Z Ran 1 test in 0.524s 2022-11-23T02:22:33.4917520Z 2022-11-23T02:22:33.4917620Z OK (skipped=1) 2022-11-23T02:22:33.4917627Z 2022-11-23T02:22:33.4917741Z Generating XML reports... 2022-11-23T02:22:33.4918200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021902.xml 2022-11-23T02:22:33.4918518Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4918891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4919058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4919442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4919618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4919624Z 2022-11-23T02:22:33.4919705Z Running tests... 2022-11-23T02:22:33.4919970Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4921121Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75052 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.512s) 2022-11-23T02:22:33.4921146Z 2022-11-23T02:22:33.4921413Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4921496Z Ran 1 test in 0.512s 2022-11-23T02:22:33.4921502Z 2022-11-23T02:22:33.4921600Z OK (skipped=1) 2022-11-23T02:22:33.4921606Z 2022-11-23T02:22:33.4921718Z Generating XML reports... 2022-11-23T02:22:33.4922175Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021906.xml 2022-11-23T02:22:33.4922498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4922877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4923049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4923486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4923672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4923678Z 2022-11-23T02:22:33.4923773Z Running tests... 2022-11-23T02:22:33.4924043Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4925066Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75139 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.509s) 2022-11-23T02:22:33.4925078Z 2022-11-23T02:22:33.4925348Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4925450Z Ran 1 test in 0.509s 2022-11-23T02:22:33.4925457Z 2022-11-23T02:22:33.4925555Z OK (skipped=1) 2022-11-23T02:22:33.4925561Z 2022-11-23T02:22:33.4925671Z Generating XML reports... 2022-11-23T02:22:33.4926135Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021909.xml 2022-11-23T02:22:33.4926455Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4926829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4926997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4927380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4927564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4927570Z 2022-11-23T02:22:33.4927667Z Running tests... 2022-11-23T02:22:33.4927938Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4928888Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.510s) 2022-11-23T02:22:33.4928895Z 2022-11-23T02:22:33.4929161Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4929259Z Ran 1 test in 0.510s 2022-11-23T02:22:33.4929314Z 2022-11-23T02:22:33.4929408Z OK (skipped=1) 2022-11-23T02:22:33.4929414Z 2022-11-23T02:22:33.4929508Z Generating XML reports... 2022-11-23T02:22:33.4929967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021913.xml 2022-11-23T02:22:33.4930288Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4930659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4930825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4931206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4931385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4931392Z 2022-11-23T02:22:33.4931488Z Running tests... 2022-11-23T02:22:33.4931751Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4932017Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4932023Z 2022-11-23T02:22:33.4932288Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4932391Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4932439Z 2022-11-23T02:22:33.4932535Z OK (skipped=1) 2022-11-23T02:22:33.4932541Z 2022-11-23T02:22:33.4932652Z Generating XML reports... 2022-11-23T02:22:33.4933113Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021916.xml 2022-11-23T02:22:33.4933426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4933802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4933968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4934354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4934535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4934541Z 2022-11-23T02:22:33.4934635Z Running tests... 2022-11-23T02:22:33.4934901Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4935163Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.4935171Z 2022-11-23T02:22:33.4935436Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4935519Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4935525Z 2022-11-23T02:22:33.4935621Z OK (skipped=1) 2022-11-23T02:22:33.4935626Z 2022-11-23T02:22:33.4935740Z Generating XML reports... 2022-11-23T02:22:33.4936197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021919.xml 2022-11-23T02:22:33.4936518Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4936892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4937059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4937441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4937626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4937632Z 2022-11-23T02:22:33.4937727Z Running tests... 2022-11-23T02:22:33.4937994Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4938263Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4938312Z 2022-11-23T02:22:33.4938583Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4938683Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4938689Z 2022-11-23T02:22:33.4938783Z OK (skipped=1) 2022-11-23T02:22:33.4938788Z 2022-11-23T02:22:33.4938899Z Generating XML reports... 2022-11-23T02:22:33.4939360Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021922.xml 2022-11-23T02:22:33.4939679Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4940053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4940216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4940598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4940778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4940788Z 2022-11-23T02:22:33.4940868Z Running tests... 2022-11-23T02:22:33.4941133Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4941396Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4941467Z 2022-11-23T02:22:33.4941735Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4941834Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4941840Z 2022-11-23T02:22:33.4941935Z OK (skipped=1) 2022-11-23T02:22:33.4941940Z 2022-11-23T02:22:33.4942053Z Generating XML reports... 2022-11-23T02:22:33.4942511Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021925.xml 2022-11-23T02:22:33.4942827Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4943199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4943371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4943755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4943941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4943947Z 2022-11-23T02:22:33.4944044Z Running tests... 2022-11-23T02:22:33.4944311Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4944570Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.4944576Z 2022-11-23T02:22:33.4944840Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4944943Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4944949Z 2022-11-23T02:22:33.4945045Z OK (skipped=1) 2022-11-23T02:22:33.4945054Z 2022-11-23T02:22:33.4945169Z Generating XML reports... 2022-11-23T02:22:33.4945627Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021928.xml 2022-11-23T02:22:33.4945944Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4946322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4946471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4946852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4947032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4947039Z 2022-11-23T02:22:33.4947137Z Running tests... 2022-11-23T02:22:33.4947404Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4947705Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4947711Z 2022-11-23T02:22:33.4947972Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4948073Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4948079Z 2022-11-23T02:22:33.4948175Z OK (skipped=1) 2022-11-23T02:22:33.4948182Z 2022-11-23T02:22:33.4948293Z Generating XML reports... 2022-11-23T02:22:33.4948750Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021931.xml 2022-11-23T02:22:33.4949066Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4949438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4949601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4949982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4950164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4950170Z 2022-11-23T02:22:33.4950266Z Running tests... 2022-11-23T02:22:33.4950576Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4950839Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4950845Z 2022-11-23T02:22:33.4951110Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4951209Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4951215Z 2022-11-23T02:22:33.4951308Z OK (skipped=1) 2022-11-23T02:22:33.4951313Z 2022-11-23T02:22:33.4951409Z Generating XML reports... 2022-11-23T02:22:33.4951862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021934.xml 2022-11-23T02:22:33.4952182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4952557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4952723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4953110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4953290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4953295Z 2022-11-23T02:22:33.4953390Z Running tests... 2022-11-23T02:22:33.4953654Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4953907Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4953914Z 2022-11-23T02:22:33.4954172Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4954276Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4954282Z 2022-11-23T02:22:33.4954375Z OK (skipped=1) 2022-11-23T02:22:33.4954381Z 2022-11-23T02:22:33.4954495Z Generating XML reports... 2022-11-23T02:22:33.4954951Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021937.xml 2022-11-23T02:22:33.4955264Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4955638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4955805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4956188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4956369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4956416Z 2022-11-23T02:22:33.4956511Z Running tests... 2022-11-23T02:22:33.4956778Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4957008Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4957031Z 2022-11-23T02:22:33.4957282Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4957380Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4957386Z 2022-11-23T02:22:33.4957482Z OK (skipped=1) 2022-11-23T02:22:33.4957488Z 2022-11-23T02:22:33.4957599Z Generating XML reports... 2022-11-23T02:22:33.4958053Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021939.xml 2022-11-23T02:22:33.4958369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4958739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4958906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4959440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4959664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4959670Z 2022-11-23T02:22:33.4959767Z Running tests... 2022-11-23T02:22:33.4960032Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4960342Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4960349Z 2022-11-23T02:22:33.4960613Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4960712Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4960718Z 2022-11-23T02:22:33.4960813Z OK (skipped=1) 2022-11-23T02:22:33.4960819Z 2022-11-23T02:22:33.4960931Z Generating XML reports... 2022-11-23T02:22:33.4961394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021942.xml 2022-11-23T02:22:33.4961714Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4962095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4962261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4962646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4962810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4962834Z 2022-11-23T02:22:33.4962912Z Running tests... 2022-11-23T02:22:33.4963182Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4963495Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56841 2022-11-23T02:22:33.4963711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56842 2022-11-23T02:22:33.4963971Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4964347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4964513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4964893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4965112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4965349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4965722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4965938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4966323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4966505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4966741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4967148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4967554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4967775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4967995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4968344Z STAGE:2022-11-23 02:19:48 56842:56842 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4968675Z STAGE:2022-11-23 02:19:48 56841:56841 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4969061Z STAGE:2022-11-23 02:19:48 56842:56842 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4969405Z STAGE:2022-11-23 02:19:48 56842:56842 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4969634Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.4969976Z STAGE:2022-11-23 02:19:48 56841:56841 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4970333Z STAGE:2022-11-23 02:19:48 56841:56841 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4970566Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.4970907Z STAGE:2022-11-23 02:19:48 56842:56842 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4971247Z STAGE:2022-11-23 02:19:48 56842:56842 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4971602Z STAGE:2022-11-23 02:19:48 56842:56842 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4971935Z STAGE:2022-11-23 02:19:48 56841:56841 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4972271Z STAGE:2022-11-23 02:19:48 56841:56841 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4972621Z STAGE:2022-11-23 02:19:48 56841:56841 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4973420Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:22:33.4973525Z warnings.warn( 2022-11-23T02:22:33.4974317Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:22:33.4974419Z warnings.warn( 2022-11-23T02:22:33.4974508Z ok (4.637s) 2022-11-23T02:22:33.4974514Z 2022-11-23T02:22:33.4974781Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4974880Z Ran 1 test in 4.637s 2022-11-23T02:22:33.4974885Z 2022-11-23T02:22:33.4974965Z OK 2022-11-23T02:22:33.4974971Z 2022-11-23T02:22:33.4975083Z Generating XML reports... 2022-11-23T02:22:33.4975546Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021945.xml 2022-11-23T02:22:33.4975909Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4976283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4976451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4976838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4977006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4977027Z 2022-11-23T02:22:33.4977108Z Running tests... 2022-11-23T02:22:33.4977377Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4977634Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.4977643Z 2022-11-23T02:22:33.4977904Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4978004Z Ran 1 test in 0.002s 2022-11-23T02:22:33.4978010Z 2022-11-23T02:22:33.4978103Z OK (skipped=1) 2022-11-23T02:22:33.4978109Z 2022-11-23T02:22:33.4978220Z Generating XML reports... 2022-11-23T02:22:33.4978712Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021953.xml 2022-11-23T02:22:33.4979029Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4979402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4979567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4979947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4980130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4980136Z 2022-11-23T02:22:33.4980230Z Running tests... 2022-11-23T02:22:33.4980496Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4980823Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57091 2022-11-23T02:22:33.4981030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57092 2022-11-23T02:22:33.4981291Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4981664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4981827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4982208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4982391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4982612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4982983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4983149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4983528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4983708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4983942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4984351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4984798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4985021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4985241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4985584Z STAGE:2022-11-23 02:19:58 57092:57092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4985916Z STAGE:2022-11-23 02:19:58 57091:57091 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4986260Z STAGE:2022-11-23 02:19:59 57092:57092 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4986614Z STAGE:2022-11-23 02:19:59 57092:57092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4986846Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.4987193Z STAGE:2022-11-23 02:19:59 57091:57091 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4987548Z STAGE:2022-11-23 02:19:59 57091:57091 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4987818Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.4988158Z STAGE:2022-11-23 02:19:59 57091:57091 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4988499Z STAGE:2022-11-23 02:19:59 57091:57091 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4988849Z STAGE:2022-11-23 02:19:59 57091:57091 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4989177Z STAGE:2022-11-23 02:19:59 57092:57092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.4989511Z STAGE:2022-11-23 02:19:59 57092:57092 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.4989866Z STAGE:2022-11-23 02:19:59 57092:57092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.4989940Z ok (4.743s) 2022-11-23T02:22:33.4989947Z 2022-11-23T02:22:33.4990219Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4990318Z Ran 1 test in 4.744s 2022-11-23T02:22:33.4990324Z 2022-11-23T02:22:33.4990402Z OK 2022-11-23T02:22:33.4990408Z 2022-11-23T02:22:33.4990520Z Generating XML reports... 2022-11-23T02:22:33.4990983Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021956.xml 2022-11-23T02:22:33.4991299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4991674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4991843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4992226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4992409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4992416Z 2022-11-23T02:22:33.4992514Z Running tests... 2022-11-23T02:22:33.4992784Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4993105Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57283 2022-11-23T02:22:33.4993318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57284 2022-11-23T02:22:33.4993577Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.4993950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4994203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4994590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4994773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4995007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.4995381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4995530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.4995914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.4996097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.4996339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.4996744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4997193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.4997416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.4997638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.4997726Z ok (4.641s) 2022-11-23T02:22:33.4997733Z 2022-11-23T02:22:33.4998004Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.4998102Z Ran 1 test in 4.641s 2022-11-23T02:22:33.4998109Z 2022-11-23T02:22:33.4998188Z OK 2022-11-23T02:22:33.4998194Z 2022-11-23T02:22:33.4998306Z Generating XML reports... 2022-11-23T02:22:33.4998769Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022004.xml 2022-11-23T02:22:33.4999089Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.4999468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.4999633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5000013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5000292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5000299Z 2022-11-23T02:22:33.5000396Z Running tests... 2022-11-23T02:22:33.5000668Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5000902Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.5000928Z 2022-11-23T02:22:33.5001175Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5001276Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5001282Z 2022-11-23T02:22:33.5001378Z OK (skipped=1) 2022-11-23T02:22:33.5001384Z 2022-11-23T02:22:33.5001498Z Generating XML reports... 2022-11-23T02:22:33.5001962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022011.xml 2022-11-23T02:22:33.5002281Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5002765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5002905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5003225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5003428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5003433Z 2022-11-23T02:22:33.5003515Z Running tests... 2022-11-23T02:22:33.5003739Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5003995Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 487 2022-11-23T02:22:33.5004170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 488 2022-11-23T02:22:33.5004388Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5004695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5004834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5005151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5005308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5005506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5005860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5005999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5006302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5006451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5006648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5006986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5007327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5007513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5007697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5007980Z STAGE:2022-11-23 02:20:17 487:487 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5008256Z STAGE:2022-11-23 02:20:17 488:488 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5008535Z STAGE:2022-11-23 02:20:17 487:487 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5008824Z STAGE:2022-11-23 02:20:17 487:487 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5009097Z STAGE:2022-11-23 02:20:17 488:488 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5009292Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5009581Z STAGE:2022-11-23 02:20:17 488:488 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5009772Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5010042Z STAGE:2022-11-23 02:20:17 487:487 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5010319Z STAGE:2022-11-23 02:20:17 487:487 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5010606Z STAGE:2022-11-23 02:20:17 487:487 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5010985Z STAGE:2022-11-23 02:20:17 488:488 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5011254Z STAGE:2022-11-23 02:20:17 488:488 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5011592Z STAGE:2022-11-23 02:20:17 488:488 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5011773Z ok (4.239s) 2022-11-23T02:22:33.5011779Z 2022-11-23T02:22:33.5011990Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5012054Z Ran 1 test in 4.240s 2022-11-23T02:22:33.5012075Z 2022-11-23T02:22:33.5012123Z OK 2022-11-23T02:22:33.5012127Z 2022-11-23T02:22:33.5012216Z Generating XML reports... 2022-11-23T02:22:33.5012572Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022014.xml 2022-11-23T02:22:33.5012816Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5013104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5013235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5013533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5013675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5013680Z 2022-11-23T02:22:33.5013788Z Running tests... 2022-11-23T02:22:33.5013997Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5014241Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 679 2022-11-23T02:22:33.5014404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 680 2022-11-23T02:22:33.5014605Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5014895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5015028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5015326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5015468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5015648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5015933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5016059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5016351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5016477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5016659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5016971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5017284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5017457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5017630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5017887Z STAGE:2022-11-23 02:20:24 679:679 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5018140Z STAGE:2022-11-23 02:20:24 680:680 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5018396Z STAGE:2022-11-23 02:20:24 679:679 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5018701Z STAGE:2022-11-23 02:20:24 679:679 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5018878Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5019134Z STAGE:2022-11-23 02:20:24 680:680 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5019401Z STAGE:2022-11-23 02:20:24 680:680 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5019579Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5019827Z STAGE:2022-11-23 02:20:24 679:679 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5020083Z STAGE:2022-11-23 02:20:24 679:679 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5020349Z STAGE:2022-11-23 02:20:24 679:679 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5020600Z STAGE:2022-11-23 02:20:24 680:680 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5020854Z STAGE:2022-11-23 02:20:24 680:680 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5021154Z STAGE:2022-11-23 02:20:24 680:680 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5021228Z ok (4.145s) 2022-11-23T02:22:33.5021233Z 2022-11-23T02:22:33.5021441Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5021519Z Ran 1 test in 4.146s 2022-11-23T02:22:33.5021524Z 2022-11-23T02:22:33.5021574Z OK 2022-11-23T02:22:33.5021593Z 2022-11-23T02:22:33.5021666Z Generating XML reports... 2022-11-23T02:22:33.5022019Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022021.xml 2022-11-23T02:22:33.5022262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5022551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5022682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5022978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5023117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5023122Z 2022-11-23T02:22:33.5023199Z Running tests... 2022-11-23T02:22:33.5023405Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5023603Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.5023608Z 2022-11-23T02:22:33.5023810Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5023889Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5023896Z 2022-11-23T02:22:33.5023971Z OK (skipped=1) 2022-11-23T02:22:33.5023975Z 2022-11-23T02:22:33.5024063Z Generating XML reports... 2022-11-23T02:22:33.5024414Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022029.xml 2022-11-23T02:22:33.5024658Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5024944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5025072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5025360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5025499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5025503Z 2022-11-23T02:22:33.5025578Z Running tests... 2022-11-23T02:22:33.5025765Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5025992Z test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.5025997Z 2022-11-23T02:22:33.5026200Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5026278Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5026284Z 2022-11-23T02:22:33.5026360Z OK (skipped=1) 2022-11-23T02:22:33.5026364Z 2022-11-23T02:22:33.5026452Z Generating XML reports... 2022-11-23T02:22:33.5026802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022031.xml 2022-11-23T02:22:33.5027045Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5027333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5027461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5027758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5027896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5027901Z 2022-11-23T02:22:33.5027977Z Running tests... 2022-11-23T02:22:33.5028215Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5028420Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:22:33.5028424Z 2022-11-23T02:22:33.5028626Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5028704Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5028708Z 2022-11-23T02:22:33.5028782Z OK (skipped=1) 2022-11-23T02:22:33.5028788Z 2022-11-23T02:22:33.5028875Z Generating XML reports... 2022-11-23T02:22:33.5029222Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022034.xml 2022-11-23T02:22:33.5029466Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5029735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5029864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5030161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5030299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5030304Z 2022-11-23T02:22:33.5030380Z Running tests... 2022-11-23T02:22:33.5030584Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5030786Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.5030791Z 2022-11-23T02:22:33.5030995Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5031075Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5031079Z 2022-11-23T02:22:33.5031156Z OK (skipped=1) 2022-11-23T02:22:33.5031160Z 2022-11-23T02:22:33.5031250Z Generating XML reports... 2022-11-23T02:22:33.5031599Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022037.xml 2022-11-23T02:22:33.5031841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5032127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5032254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5032545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5032683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5032721Z 2022-11-23T02:22:33.5032794Z Running tests... 2022-11-23T02:22:33.5033000Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5033237Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1106 2022-11-23T02:22:33.5033402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1107 2022-11-23T02:22:33.5033605Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5033875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5034006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5034300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5034443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5034625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5034910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5035070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5035364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5035501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5035684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5036000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5036312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5036484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5036653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5036913Z STAGE:2022-11-23 02:20:43 1107:1107 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5037174Z STAGE:2022-11-23 02:20:43 1106:1106 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5037437Z STAGE:2022-11-23 02:20:43 1106:1106 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5037708Z STAGE:2022-11-23 02:20:43 1106:1106 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5037886Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5038146Z STAGE:2022-11-23 02:20:43 1107:1107 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5038414Z STAGE:2022-11-23 02:20:43 1107:1107 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5038586Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5038844Z STAGE:2022-11-23 02:20:43 1106:1106 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5039088Z STAGE:2022-11-23 02:20:43 1106:1106 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5039357Z STAGE:2022-11-23 02:20:43 1106:1106 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5039611Z STAGE:2022-11-23 02:20:43 1107:1107 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5039866Z STAGE:2022-11-23 02:20:43 1107:1107 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5040224Z STAGE:2022-11-23 02:20:43 1107:1107 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5040295Z ok (4.456s) 2022-11-23T02:22:33.5040301Z 2022-11-23T02:22:33.5040689Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5040776Z Ran 1 test in 4.457s 2022-11-23T02:22:33.5040781Z 2022-11-23T02:22:33.5040853Z OK 2022-11-23T02:22:33.5040859Z 2022-11-23T02:22:33.5040954Z Generating XML reports... 2022-11-23T02:22:33.5041340Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022040.xml 2022-11-23T02:22:33.5041605Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5041917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5042056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5042380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5042534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5042539Z 2022-11-23T02:22:33.5042623Z Running tests... 2022-11-23T02:22:33.5042891Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5043157Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1298 2022-11-23T02:22:33.5043328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1299 2022-11-23T02:22:33.5043549Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5043861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5043989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5044307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5044458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5044659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5044969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5045107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5045425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5045576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5045773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5046111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5046444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5046628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5046813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5047095Z STAGE:2022-11-23 02:20:50 1298:1298 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5047373Z STAGE:2022-11-23 02:20:50 1299:1299 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5047653Z STAGE:2022-11-23 02:20:50 1298:1298 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5047944Z STAGE:2022-11-23 02:20:50 1298:1298 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5048176Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5048456Z STAGE:2022-11-23 02:20:50 1299:1299 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5048747Z STAGE:2022-11-23 02:20:50 1299:1299 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5048934Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5049207Z STAGE:2022-11-23 02:20:50 1298:1298 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5049491Z STAGE:2022-11-23 02:20:50 1298:1298 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5049766Z STAGE:2022-11-23 02:20:50 1298:1298 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5050042Z STAGE:2022-11-23 02:20:50 1299:1299 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5050323Z STAGE:2022-11-23 02:20:50 1299:1299 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5050611Z STAGE:2022-11-23 02:20:50 1299:1299 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5050690Z ok (4.250s) 2022-11-23T02:22:33.5050728Z 2022-11-23T02:22:33.5050957Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5051152Z Ran 1 test in 4.251s 2022-11-23T02:22:33.5051157Z 2022-11-23T02:22:33.5051224Z OK 2022-11-23T02:22:33.5051229Z 2022-11-23T02:22:33.5051320Z Generating XML reports... 2022-11-23T02:22:33.5051695Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022048.xml 2022-11-23T02:22:33.5052053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5052338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5052467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5052758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5052899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5052904Z 2022-11-23T02:22:33.5052977Z Running tests... 2022-11-23T02:22:33.5053186Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5053381Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.5053386Z 2022-11-23T02:22:33.5053589Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5053668Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5053672Z 2022-11-23T02:22:33.5053746Z OK (skipped=1) 2022-11-23T02:22:33.5053753Z 2022-11-23T02:22:33.5053840Z Generating XML reports... 2022-11-23T02:22:33.5054174Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022055.xml 2022-11-23T02:22:33.5054565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5054852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5054980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5055275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5055414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5055419Z 2022-11-23T02:22:33.5055493Z Running tests... 2022-11-23T02:22:33.5055695Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5055922Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:22:33.5055927Z 2022-11-23T02:22:33.5056133Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5056210Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5056214Z 2022-11-23T02:22:33.5056290Z OK (skipped=1) 2022-11-23T02:22:33.5056295Z 2022-11-23T02:22:33.5056379Z Generating XML reports... 2022-11-23T02:22:33.5056727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022058.xml 2022-11-23T02:22:33.5056968Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5057252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5057379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5057667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5057808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5057813Z 2022-11-23T02:22:33.5057886Z Running tests... 2022-11-23T02:22:33.5058120Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5058424Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:22:33.5058429Z 2022-11-23T02:22:33.5058632Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5058696Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5058713Z 2022-11-23T02:22:33.5058774Z OK (skipped=1) 2022-11-23T02:22:33.5058778Z 2022-11-23T02:22:33.5058862Z Generating XML reports... 2022-11-23T02:22:33.5059208Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022101.xml 2022-11-23T02:22:33.5059451Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5059734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5059861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5060156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5060294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5060299Z 2022-11-23T02:22:33.5060374Z Running tests... 2022-11-23T02:22:33.5060577Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5060772Z test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:22:33.5060778Z 2022-11-23T02:22:33.5060977Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5061058Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5061062Z 2022-11-23T02:22:33.5061135Z OK (skipped=1) 2022-11-23T02:22:33.5061139Z 2022-11-23T02:22:33.5061226Z Generating XML reports... 2022-11-23T02:22:33.5061574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022104.xml 2022-11-23T02:22:33.5061815Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5062098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5062225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5062517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5062653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5062692Z 2022-11-23T02:22:33.5062766Z Running tests... 2022-11-23T02:22:33.5062957Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5063171Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.001s) 2022-11-23T02:22:33.5063176Z 2022-11-23T02:22:33.5063376Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5063454Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5063459Z 2022-11-23T02:22:33.5063533Z OK (skipped=1) 2022-11-23T02:22:33.5063537Z 2022-11-23T02:22:33.5063623Z Generating XML reports... 2022-11-23T02:22:33.5063975Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022106.xml 2022-11-23T02:22:33.5064217Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5064501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5064630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5064921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5065111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5065115Z 2022-11-23T02:22:33.5065190Z Running tests... 2022-11-23T02:22:33.5065394Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5065627Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.001s) 2022-11-23T02:22:33.5065632Z 2022-11-23T02:22:33.5065883Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5065963Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5065967Z 2022-11-23T02:22:33.5066040Z OK (skipped=1) 2022-11-23T02:22:33.5066047Z 2022-11-23T02:22:33.5066133Z Generating XML reports... 2022-11-23T02:22:33.5066478Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022109.xml 2022-11-23T02:22:33.5066718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5067003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5067128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5067404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5067543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5067548Z 2022-11-23T02:22:33.5067621Z Running tests... 2022-11-23T02:22:33.5067823Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5068052Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.001s) 2022-11-23T02:22:33.5068057Z 2022-11-23T02:22:33.5068259Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5068335Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5068339Z 2022-11-23T02:22:33.5068415Z OK (skipped=1) 2022-11-23T02:22:33.5068419Z 2022-11-23T02:22:33.5068507Z Generating XML reports... 2022-11-23T02:22:33.5068852Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022112.xml 2022-11-23T02:22:33.5069093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5069375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5069501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5069827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5069963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5069968Z 2022-11-23T02:22:33.5070041Z Running tests... 2022-11-23T02:22:33.5070247Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5070464Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:22:33.5070469Z 2022-11-23T02:22:33.5070668Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5070744Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5070748Z 2022-11-23T02:22:33.5070823Z OK (skipped=1) 2022-11-23T02:22:33.5070827Z 2022-11-23T02:22:33.5070912Z Generating XML reports... 2022-11-23T02:22:33.5071258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022115.xml 2022-11-23T02:22:33.5071484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5071763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5071924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5072216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5072352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5072356Z 2022-11-23T02:22:33.5072429Z Running tests... 2022-11-23T02:22:33.5072631Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5072864Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1954 2022-11-23T02:22:33.5073025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1955 2022-11-23T02:22:33.5073228Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5073515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5073641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5073930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5074072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5074254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5074535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5074662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5074951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5075089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5075273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5075582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5075888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5076057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5076212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5076316Z ok (4.257s) 2022-11-23T02:22:33.5076321Z 2022-11-23T02:22:33.5076525Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5076603Z Ran 1 test in 4.257s 2022-11-23T02:22:33.5076608Z 2022-11-23T02:22:33.5076671Z OK 2022-11-23T02:22:33.5076676Z 2022-11-23T02:22:33.5076762Z Generating XML reports... 2022-11-23T02:22:33.5077108Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022118.xml 2022-11-23T02:22:33.5077349Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5077633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5077759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5078050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5078191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5078196Z 2022-11-23T02:22:33.5078270Z Running tests... 2022-11-23T02:22:33.5078473Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5078759Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2140 2022-11-23T02:22:33.5078923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2141 2022-11-23T02:22:33.5079123Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5079409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5079538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5079828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5079972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5080211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5080661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5080800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5081118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5081269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5081467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5081806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5082142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5082327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5082513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5082796Z STAGE:2022-11-23 02:21:28 2141:2141 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5083073Z STAGE:2022-11-23 02:21:28 2140:2140 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5083355Z STAGE:2022-11-23 02:21:28 2140:2140 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5083648Z STAGE:2022-11-23 02:21:28 2140:2140 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5083837Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5084163Z STAGE:2022-11-23 02:21:28 2141:2141 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5084449Z STAGE:2022-11-23 02:21:28 2141:2141 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5084635Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 2 (function operator()) 2022-11-23T02:22:33.5084710Z ok (4.454s) 2022-11-23T02:22:33.5084716Z 2022-11-23T02:22:33.5084937Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5085019Z Ran 1 test in 4.454s 2022-11-23T02:22:33.5085024Z 2022-11-23T02:22:33.5085091Z OK 2022-11-23T02:22:33.5085096Z 2022-11-23T02:22:33.5085191Z Generating XML reports... 2022-11-23T02:22:33.5085554Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022125.xml 2022-11-23T02:22:33.5085817Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5086128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5086265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5086625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5086775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5086781Z 2022-11-23T02:22:33.5086863Z Running tests... 2022-11-23T02:22:33.5087087Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5087355Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2332 2022-11-23T02:22:33.5087526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2333 2022-11-23T02:22:33.5087747Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5088053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5088194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5088513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5088663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5088860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5089169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5089307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5089626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5089775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5089972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5090312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5090650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5090819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5091108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5091386Z STAGE:2022-11-23 02:21:35 2333:2333 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5091794Z STAGE:2022-11-23 02:21:35 2332:2332 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:22:33.5092055Z STAGE:2022-11-23 02:21:36 2332:2332 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5092328Z STAGE:2022-11-23 02:21:36 2332:2332 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5092502Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5092761Z STAGE:2022-11-23 02:21:36 2333:2333 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:22:33.5093025Z STAGE:2022-11-23 02:21:36 2333:2333 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:22:33.5093195Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:22:33.5093267Z ok (4.741s) 2022-11-23T02:22:33.5093273Z 2022-11-23T02:22:33.5093482Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5093562Z Ran 1 test in 4.742s 2022-11-23T02:22:33.5093567Z 2022-11-23T02:22:33.5093632Z OK 2022-11-23T02:22:33.5093636Z 2022-11-23T02:22:33.5093725Z Generating XML reports... 2022-11-23T02:22:33.5094108Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022133.xml 2022-11-23T02:22:33.5094356Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5094643Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5094771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5095067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5095208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5095215Z 2022-11-23T02:22:33.5095293Z Running tests... 2022-11-23T02:22:33.5095485Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5095706Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:22:33.5095728Z 2022-11-23T02:22:33.5095920Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5096000Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5096005Z 2022-11-23T02:22:33.5096080Z OK (skipped=1) 2022-11-23T02:22:33.5096085Z 2022-11-23T02:22:33.5096173Z Generating XML reports... 2022-11-23T02:22:33.5096527Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022141.xml 2022-11-23T02:22:33.5096771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5097058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5097190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5097482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5097624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5097629Z 2022-11-23T02:22:33.5097704Z Running tests... 2022-11-23T02:22:33.5097909Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5098114Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:22:33.5098119Z 2022-11-23T02:22:33.5098322Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5098402Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5098406Z 2022-11-23T02:22:33.5098482Z OK (skipped=1) 2022-11-23T02:22:33.5098524Z 2022-11-23T02:22:33.5098617Z Generating XML reports... 2022-11-23T02:22:33.5098973Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022143.xml 2022-11-23T02:22:33.5099215Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5099506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5099635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5099914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5106183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5106194Z 2022-11-23T02:22:33.5106293Z Running tests... 2022-11-23T02:22:33.5106528Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5106758Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:22:33.5106763Z 2022-11-23T02:22:33.5106959Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5107026Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5107031Z 2022-11-23T02:22:33.5107194Z OK (skipped=1) 2022-11-23T02:22:33.5107199Z 2022-11-23T02:22:33.5107276Z Generating XML reports... 2022-11-23T02:22:33.5107632Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022146.xml 2022-11-23T02:22:33.5107863Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5108149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5108273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5108567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5108707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5108712Z 2022-11-23T02:22:33.5108784Z Running tests... 2022-11-23T02:22:33.5108993Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5109218Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:22:33.5109223Z 2022-11-23T02:22:33.5109425Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5109505Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5109510Z 2022-11-23T02:22:33.5109580Z OK (skipped=1) 2022-11-23T02:22:33.5109584Z 2022-11-23T02:22:33.5109668Z Generating XML reports... 2022-11-23T02:22:33.5110020Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022149.xml 2022-11-23T02:22:33.5110264Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5110546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5110671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5110952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5111085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5111089Z 2022-11-23T02:22:33.5111160Z Running tests... 2022-11-23T02:22:33.5111363Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5111571Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.001s) 2022-11-23T02:22:33.5111576Z 2022-11-23T02:22:33.5111815Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5111888Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5111893Z 2022-11-23T02:22:33.5111963Z OK (skipped=1) 2022-11-23T02:22:33.5111967Z 2022-11-23T02:22:33.5112047Z Generating XML reports... 2022-11-23T02:22:33.5112393Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022152.xml 2022-11-23T02:22:33.5112636Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5112919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5113043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5113335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5113473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5113480Z 2022-11-23T02:22:33.5113552Z Running tests... 2022-11-23T02:22:33.5113754Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5113967Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.001s) 2022-11-23T02:22:33.5114065Z 2022-11-23T02:22:33.5114271Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5114351Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5114356Z 2022-11-23T02:22:33.5114426Z OK (skipped=1) 2022-11-23T02:22:33.5114430Z 2022-11-23T02:22:33.5114513Z Generating XML reports... 2022-11-23T02:22:33.5114866Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022155.xml 2022-11-23T02:22:33.5115093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5115378Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5115507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5115802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5115941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5115946Z 2022-11-23T02:22:33.5116016Z Running tests... 2022-11-23T02:22:33.5116217Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5116454Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2872 2022-11-23T02:22:33.5116611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2873 2022-11-23T02:22:33.5116808Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5117090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5117213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5117502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5117637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5117813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5118098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5118225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5118514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5118686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5118865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5119181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5119485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5119647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5119802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5119866Z ok (6.257s) 2022-11-23T02:22:33.5119871Z 2022-11-23T02:22:33.5120074Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5120215Z Ran 1 test in 6.257s 2022-11-23T02:22:33.5120223Z 2022-11-23T02:22:33.5120285Z OK 2022-11-23T02:22:33.5120290Z 2022-11-23T02:22:33.5120539Z Generating XML reports... 2022-11-23T02:22:33.5120918Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022158.xml 2022-11-23T02:22:33.5121224Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5121535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5121671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5121983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5122126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5122131Z 2022-11-23T02:22:33.5122208Z Running tests... 2022-11-23T02:22:33.5122428Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5122648Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s) 2022-11-23T02:22:33.5122653Z 2022-11-23T02:22:33.5122868Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5122949Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5122955Z 2022-11-23T02:22:33.5123030Z OK (skipped=1) 2022-11-23T02:22:33.5123036Z 2022-11-23T02:22:33.5123127Z Generating XML reports... 2022-11-23T02:22:33.5123503Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022207.xml 2022-11-23T02:22:33.5123762Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5124068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5124190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5124503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5124649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5124654Z 2022-11-23T02:22:33.5124733Z Running tests... 2022-11-23T02:22:33.5124955Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5125204Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3126 2022-11-23T02:22:33.5125374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3127 2022-11-23T02:22:33.5125586Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5125890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5126091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5126411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5126557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5126754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5127057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5127189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5127500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5127648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5127842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5128179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5128547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5128732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5128910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5129113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7l9nklk6 2022-11-23T02:22:33.5129315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7l9nklk6/_remote_module_non_scriptable.py 2022-11-23T02:22:33.5129510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n6x7cni 2022-11-23T02:22:33.5129715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n6x7cni/_remote_module_non_scriptable.py 2022-11-23T02:22:33.5129790Z ok (4.638s) 2022-11-23T02:22:33.5129795Z 2022-11-23T02:22:33.5130020Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5130100Z Ran 1 test in 4.639s 2022-11-23T02:22:33.5130105Z 2022-11-23T02:22:33.5130172Z OK 2022-11-23T02:22:33.5130177Z 2022-11-23T02:22:33.5130267Z Generating XML reports... 2022-11-23T02:22:33.5130647Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022210.xml 2022-11-23T02:22:33.5130908Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5131317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5131448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5131755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5131901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5131906Z 2022-11-23T02:22:33.5132083Z Running tests... 2022-11-23T02:22:33.5132288Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5132547Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3312 2022-11-23T02:22:33.5132701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3313 2022-11-23T02:22:33.5132896Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:22:33.5133177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5133301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5133627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5133752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5133931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:22:33.5134211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5134335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5134623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5134759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5134934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:22:33.5135240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5135545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:22:33.5135743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:22:33.5135909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:22:33.5136097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx1ovh_g 2022-11-23T02:22:33.5136295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx1ovh_g/_remote_module_non_scriptable.py 2022-11-23T02:22:33.5136478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb45gw4rc 2022-11-23T02:22:33.5136671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb45gw4rc/_remote_module_non_scriptable.py 2022-11-23T02:22:33.5137269Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:22:33.5137855Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:22:33.5137928Z ok (6.448s) 2022-11-23T02:22:33.5137933Z 2022-11-23T02:22:33.5138141Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5138218Z Ran 1 test in 6.448s 2022-11-23T02:22:33.5138225Z 2022-11-23T02:22:33.5138285Z OK 2022-11-23T02:22:33.5138289Z 2022-11-23T02:22:33.5138372Z Generating XML reports... 2022-11-23T02:22:33.5138724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022218.xml 2022-11-23T02:22:33.5138962Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5139249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5139376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5139703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5139840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5139845Z 2022-11-23T02:22:33.5139917Z Running tests... 2022-11-23T02:22:33.5140124Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5140482Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:22:33.5140488Z 2022-11-23T02:22:33.5140689Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5140753Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5140770Z 2022-11-23T02:22:33.5140830Z OK (skipped=1) 2022-11-23T02:22:33.5140834Z 2022-11-23T02:22:33.5140919Z Generating XML reports... 2022-11-23T02:22:33.5141268Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022227.xml 2022-11-23T02:22:33.5141508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:22:33.5141793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:22:33.5141981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:22:33.5142275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:22:33.5142413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:22:33.5142418Z 2022-11-23T02:22:33.5142490Z Running tests... 2022-11-23T02:22:33.5142694Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5143052Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:22:33.5143059Z 2022-11-23T02:22:33.5143260Z ---------------------------------------------------------------------- 2022-11-23T02:22:33.5143332Z Ran 1 test in 0.002s 2022-11-23T02:22:33.5143337Z 2022-11-23T02:22:33.5143410Z OK (skipped=1) 2022-11-23T02:22:33.5143414Z 2022-11-23T02:22:33.5143501Z Generating XML reports... 2022-11-23T02:22:33.5143847Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022230.xml 2022-11-23T02:22:33.5143853Z 2022-11-23T02:22:33.5144230Z ##[endgroup] 2022-11-23T02:22:33.5144598Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_6u__3oec) 2022-11-23T02:22:33.5144604Z 2022-11-23T02:22:33.5144695Z Shard 2: gloo should be run in 1 2022-11-23T02:22:33.5144885Z Running distributed/test_store ... [2022-11-23 02:22:33.310011] 2022-11-23T02:22:33.5145321Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:22:33.310374] 2022-11-23T02:24:24.2590392Z 2022-11-23T02:24:24.2594720Z Expand the folded group to see the log file of distributed/test_store 2022-11-23T02:24:24.2597568Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_store_a3op8ur0) 2022-11-23T02:24:24.2600861Z , <__main__.FileStoreTest testMethod=test_init_pg_and_rpc_with_same_file>, <__main__.FileStoreTest testMethod=test_refcount>, <__main__.FileStoreTest testMethod=test_set_get>]> 2022-11-23T02:24:24.2603239Z test_compare_set (__main__.FileStoreTest) 2022-11-23T02:24:24.2604164Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) 2022-11-23T02:24:24.2605060Z test_refcount (__main__.FileStoreTest) 2022-11-23T02:24:24.2606385Z test_set_get (__main__.FileStoreTest) 2022-11-23T02:24:24.2608745Z , <__main__.HashStoreTest testMethod=test_set_get>]> 2022-11-23T02:24:24.2610763Z test_compare_set (__main__.HashStoreTest) 2022-11-23T02:24:24.2611544Z test_set_get (__main__.HashStoreTest) 2022-11-23T02:24:24.2612871Z , <__main__.PrefixFileStoreTest testMethod=test_set_get>]> 2022-11-23T02:24:24.2614231Z test_compare_set (__main__.PrefixFileStoreTest) 2022-11-23T02:24:24.2615637Z test_set_get (__main__.PrefixFileStoreTest) 2022-11-23T02:24:24.2617481Z ]> 2022-11-23T02:24:24.2618622Z test_get_underlying_store (__main__.PrefixStoreTest) 2022-11-23T02:24:24.2619965Z , <__main__.PrefixTCPStoreTest testMethod=test_set_get>]> 2022-11-23T02:24:24.2621265Z test_compare_set (__main__.PrefixTCPStoreTest) 2022-11-23T02:24:24.2622121Z test_set_get (__main__.PrefixTCPStoreTest) 2022-11-23T02:24:24.2623201Z ]> 2022-11-23T02:24:24.2624203Z test_set_get (__main__.PythonStoreTest) 2022-11-23T02:24:24.2625597Z ]> 2022-11-23T02:24:24.2627004Z test_nominal (__main__.RendezvousEnvTest) 2022-11-23T02:24:24.2629012Z , <__main__.RendezvousFileTest testMethod=test_nominal>]> 2022-11-23T02:24:24.2630805Z test_common_errors (__main__.RendezvousFileTest) 2022-11-23T02:24:24.2632077Z test_nominal (__main__.RendezvousFileTest) 2022-11-23T02:24:24.2634686Z , <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]> 2022-11-23T02:24:24.2637973Z test_common_errors (__main__.RendezvousTCPTest) 2022-11-23T02:24:24.2639217Z test_dns_timeout (__main__.RendezvousTCPTest) 2022-11-23T02:24:24.2640698Z test_nominal (__main__.RendezvousTCPTest) 2022-11-23T02:24:24.2641978Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) 2022-11-23T02:24:24.2644070Z , <__main__.RendezvousTest testMethod=test_url_with_node_params>]> 2022-11-23T02:24:24.2645917Z test_unknown_handler (__main__.RendezvousTest) 2022-11-23T02:24:24.2647227Z test_url_with_node_params (__main__.RendezvousTest) 2022-11-23T02:24:24.2651089Z , <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]> 2022-11-23T02:24:24.2653613Z test_address_already_in_use (__main__.TCPStoreTest) 2022-11-23T02:24:24.2654454Z test_compare_set (__main__.TCPStoreTest) 2022-11-23T02:24:24.2655361Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) 2022-11-23T02:24:24.2656341Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) 2022-11-23T02:24:24.2657338Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) 2022-11-23T02:24:24.2658247Z test_multitenancy (__main__.TCPStoreTest) 2022-11-23T02:24:24.2659057Z test_numkeys_delkeys (__main__.TCPStoreTest) 2022-11-23T02:24:24.2659840Z test_set_get (__main__.TCPStoreTest) 2022-11-23T02:24:24.2661633Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2663368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2664557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2666162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2667403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2667995Z 2022-11-23T02:24:24.2668237Z Running tests... 2022-11-23T02:24:24.2669362Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2670397Z test_compare_set (__main__.FileStoreTest) ... ok (0.490s) 2022-11-23T02:24:24.2670938Z 2022-11-23T02:24:24.2671661Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2672493Z Ran 1 test in 0.491s 2022-11-23T02:24:24.2672890Z 2022-11-23T02:24:24.2673092Z OK 2022-11-23T02:24:24.2673403Z 2022-11-23T02:24:24.2673675Z Generating XML reports... 2022-11-23T02:24:24.2675221Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022236.xml 2022-11-23T02:24:24.2677023Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2678729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2679922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2681652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2682903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2683496Z 2022-11-23T02:24:24.2683721Z Running tests... 2022-11-23T02:24:24.2684809Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2686291Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:24:24.2688359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:24:24.2689400Z ok (0.548s) 2022-11-23T02:24:24.2689754Z 2022-11-23T02:24:24.2690482Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2691317Z Ran 1 test in 0.549s 2022-11-23T02:24:24.2691714Z 2022-11-23T02:24:24.2691891Z OK 2022-11-23T02:24:24.2692199Z 2022-11-23T02:24:24.2692491Z Generating XML reports... 2022-11-23T02:24:24.2694005Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022239.xml 2022-11-23T02:24:24.2695668Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2697506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2699588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2702326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2703803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2704397Z 2022-11-23T02:24:24.2704643Z Running tests... 2022-11-23T02:24:24.2705772Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2706777Z test_refcount (__main__.FileStoreTest) ... ok (0.495s) 2022-11-23T02:24:24.2707218Z 2022-11-23T02:24:24.2707633Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2708109Z Ran 1 test in 0.495s 2022-11-23T02:24:24.2708336Z 2022-11-23T02:24:24.2708453Z OK 2022-11-23T02:24:24.2708634Z 2022-11-23T02:24:24.2708938Z Generating XML reports... 2022-11-23T02:24:24.2709797Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022242.xml 2022-11-23T02:24:24.2710734Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2711690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2712355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2713244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2713950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2714277Z 2022-11-23T02:24:24.2714403Z Running tests... 2022-11-23T02:24:24.2715025Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2715588Z test_set_get (__main__.FileStoreTest) ... ok (0.495s) 2022-11-23T02:24:24.2715882Z 2022-11-23T02:24:24.2716286Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2716763Z Ran 1 test in 0.495s 2022-11-23T02:24:24.2716984Z 2022-11-23T02:24:24.2717100Z OK 2022-11-23T02:24:24.2717277Z 2022-11-23T02:24:24.2717441Z Generating XML reports... 2022-11-23T02:24:24.2718351Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022246.xml 2022-11-23T02:24:24.2719292Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2720325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2720989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2721873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2722578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2723035Z 2022-11-23T02:24:24.2723136Z Running tests... 2022-11-23T02:24:24.2723559Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2723966Z test_compare_set (__main__.HashStoreTest) ... ok (0.485s) 2022-11-23T02:24:24.2724177Z 2022-11-23T02:24:24.2724465Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2724803Z Ran 1 test in 0.486s 2022-11-23T02:24:24.2724961Z 2022-11-23T02:24:24.2725044Z OK 2022-11-23T02:24:24.2725172Z 2022-11-23T02:24:24.2725290Z Generating XML reports... 2022-11-23T02:24:24.2725891Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022249.xml 2022-11-23T02:24:24.2726529Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2727192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2727667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2728289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2728786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2729023Z 2022-11-23T02:24:24.2729125Z Running tests... 2022-11-23T02:24:24.2729559Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2729940Z test_set_get (__main__.HashStoreTest) ... ok (0.484s) 2022-11-23T02:24:24.2730144Z 2022-11-23T02:24:24.2730426Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2730763Z Ran 1 test in 0.484s 2022-11-23T02:24:24.2730919Z 2022-11-23T02:24:24.2731003Z OK 2022-11-23T02:24:24.2731130Z 2022-11-23T02:24:24.2731247Z Generating XML reports... 2022-11-23T02:24:24.2731845Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022252.xml 2022-11-23T02:24:24.2732570Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2733219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2733690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2734315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2734803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2735038Z 2022-11-23T02:24:24.2735139Z Running tests... 2022-11-23T02:24:24.2735570Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2735990Z test_compare_set (__main__.PrefixFileStoreTest) ... ok (0.484s) 2022-11-23T02:24:24.2736203Z 2022-11-23T02:24:24.2736486Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2736827Z Ran 1 test in 0.484s 2022-11-23T02:24:24.2736987Z 2022-11-23T02:24:24.2737072Z OK 2022-11-23T02:24:24.2737200Z 2022-11-23T02:24:24.2737318Z Generating XML reports... 2022-11-23T02:24:24.2738004Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022256.xml 2022-11-23T02:24:24.2738685Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2739343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2739794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2740411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2740906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2741137Z 2022-11-23T02:24:24.2741242Z Running tests... 2022-11-23T02:24:24.2741675Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2742088Z test_set_get (__main__.PrefixFileStoreTest) ... ok (0.483s) 2022-11-23T02:24:24.2742305Z 2022-11-23T02:24:24.2742588Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2742914Z Ran 1 test in 0.483s 2022-11-23T02:24:24.2743071Z 2022-11-23T02:24:24.2743156Z OK 2022-11-23T02:24:24.2743286Z 2022-11-23T02:24:24.2743401Z Generating XML reports... 2022-11-23T02:24:24.2744026Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022259.xml 2022-11-23T02:24:24.2744697Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2745355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2745821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2746431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2746925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2747155Z 2022-11-23T02:24:24.2747257Z Running tests... 2022-11-23T02:24:24.2747694Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2748120Z test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.014s) 2022-11-23T02:24:24.2748352Z 2022-11-23T02:24:24.2748636Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2748971Z Ran 1 test in 0.014s 2022-11-23T02:24:24.2749114Z 2022-11-23T02:24:24.2749196Z OK 2022-11-23T02:24:24.2749322Z 2022-11-23T02:24:24.2749439Z Generating XML reports... 2022-11-23T02:24:24.2750044Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123022302.xml 2022-11-23T02:24:24.2750758Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2751418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2751888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2752509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2752991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2753218Z 2022-11-23T02:24:24.2753318Z Running tests... 2022-11-23T02:24:24.2753749Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2754166Z test_compare_set (__main__.PrefixTCPStoreTest) ... ok (0.495s) 2022-11-23T02:24:24.2754392Z 2022-11-23T02:24:24.2754675Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2755016Z Ran 1 test in 0.495s 2022-11-23T02:24:24.2755172Z 2022-11-23T02:24:24.2755241Z OK 2022-11-23T02:24:24.2755369Z 2022-11-23T02:24:24.2755486Z Generating XML reports... 2022-11-23T02:24:24.2756104Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022305.xml 2022-11-23T02:24:24.2756821Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2757487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2757959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2758578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2759072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2759287Z 2022-11-23T02:24:24.2759390Z Running tests... 2022-11-23T02:24:24.2759826Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2760296Z test_set_get (__main__.PrefixTCPStoreTest) ... ok (0.484s) 2022-11-23T02:24:24.2760511Z 2022-11-23T02:24:24.2760793Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2761122Z Ran 1 test in 0.484s 2022-11-23T02:24:24.2761279Z 2022-11-23T02:24:24.2761366Z OK 2022-11-23T02:24:24.2761492Z 2022-11-23T02:24:24.2761596Z Generating XML reports... 2022-11-23T02:24:24.2762215Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022308.xml 2022-11-23T02:24:24.2762880Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2763541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2764008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2764625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2765117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2765348Z 2022-11-23T02:24:24.2765448Z Running tests... 2022-11-23T02:24:24.2765872Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2766275Z test_set_get (__main__.PythonStoreTest) ... ok (0.480s) 2022-11-23T02:24:24.2766484Z 2022-11-23T02:24:24.2766766Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2767100Z Ran 1 test in 0.481s 2022-11-23T02:24:24.2767257Z 2022-11-23T02:24:24.2767342Z OK 2022-11-23T02:24:24.2767468Z 2022-11-23T02:24:24.2767585Z Generating XML reports... 2022-11-23T02:24:24.2768169Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123022312.xml 2022-11-23T02:24:24.2768825Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2769541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2770007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2770629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2771121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2771349Z 2022-11-23T02:24:24.2771447Z Running tests... 2022-11-23T02:24:24.2771870Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2772271Z test_nominal (__main__.RendezvousEnvTest) ... ok (0.492s) 2022-11-23T02:24:24.2772483Z 2022-11-23T02:24:24.2772767Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2773105Z Ran 1 test in 0.492s 2022-11-23T02:24:24.2773262Z 2022-11-23T02:24:24.2773345Z OK 2022-11-23T02:24:24.2773474Z 2022-11-23T02:24:24.2773592Z Generating XML reports... 2022-11-23T02:24:24.2774198Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123022315.xml 2022-11-23T02:24:24.2774851Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2775559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2776027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2776650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2777139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2777369Z 2022-11-23T02:24:24.2777467Z Running tests... 2022-11-23T02:24:24.2777900Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2778308Z test_common_errors (__main__.RendezvousFileTest) ... ok (0.481s) 2022-11-23T02:24:24.2778529Z 2022-11-23T02:24:24.2778806Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2779139Z Ran 1 test in 0.482s 2022-11-23T02:24:24.2779299Z 2022-11-23T02:24:24.2779382Z OK 2022-11-23T02:24:24.2779510Z 2022-11-23T02:24:24.2779633Z Generating XML reports... 2022-11-23T02:24:24.2780247Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022318.xml 2022-11-23T02:24:24.2780917Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2781567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2782037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2795699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2796237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2796479Z 2022-11-23T02:24:24.2796568Z Running tests... 2022-11-23T02:24:24.2797028Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2797452Z test_nominal (__main__.RendezvousFileTest) ... ok (0.484s) 2022-11-23T02:24:24.2797672Z 2022-11-23T02:24:24.2797963Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2798302Z Ran 1 test in 0.484s 2022-11-23T02:24:24.2798464Z 2022-11-23T02:24:24.2798551Z OK 2022-11-23T02:24:24.2798683Z 2022-11-23T02:24:24.2798806Z Generating XML reports... 2022-11-23T02:24:24.2799417Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022322.xml 2022-11-23T02:24:24.2800089Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2800832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2801532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2802168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2802671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2802904Z 2022-11-23T02:24:24.2803006Z Running tests... 2022-11-23T02:24:24.2803429Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2803851Z test_common_errors (__main__.RendezvousTCPTest) ... ok (0.482s) 2022-11-23T02:24:24.2804076Z 2022-11-23T02:24:24.2804364Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2804711Z Ran 1 test in 0.482s 2022-11-23T02:24:24.2804953Z 2022-11-23T02:24:24.2805038Z OK 2022-11-23T02:24:24.2805170Z 2022-11-23T02:24:24.2805292Z Generating XML reports... 2022-11-23T02:24:24.2805916Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022325.xml 2022-11-23T02:24:24.2806589Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2807315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2807792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2808421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2808904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2809140Z 2022-11-23T02:24:24.2809242Z Running tests... 2022-11-23T02:24:24.2809686Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2810538Z test_dns_timeout (__main__.RendezvousTCPTest) ... [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -3 - Temporary failure in name resolution). 2022-11-23T02:24:24.2811193Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456). 2022-11-23T02:24:24.2811566Z ok (0.496s) 2022-11-23T02:24:24.2811711Z 2022-11-23T02:24:24.2812003Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2812333Z Ran 1 test in 0.496s 2022-11-23T02:24:24.2812492Z 2022-11-23T02:24:24.2812577Z OK 2022-11-23T02:24:24.2812705Z 2022-11-23T02:24:24.2812824Z Generating XML reports... 2022-11-23T02:24:24.2813447Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022328.xml 2022-11-23T02:24:24.2814116Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2814783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2815260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2815868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2816362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2816604Z 2022-11-23T02:24:24.2816707Z Running tests... 2022-11-23T02:24:24.2817149Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2817560Z test_nominal (__main__.RendezvousTCPTest) ... ok (0.492s) 2022-11-23T02:24:24.2817777Z 2022-11-23T02:24:24.2818061Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2818393Z Ran 1 test in 0.492s 2022-11-23T02:24:24.2818554Z 2022-11-23T02:24:24.2818622Z OK 2022-11-23T02:24:24.2818751Z 2022-11-23T02:24:24.2818870Z Generating XML reports... 2022-11-23T02:24:24.2819492Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022332.xml 2022-11-23T02:24:24.2820238Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2820906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2821381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2822010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2822493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2822725Z 2022-11-23T02:24:24.2822827Z Running tests... 2022-11-23T02:24:24.2823263Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2823701Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... ok (10.681s) 2022-11-23T02:24:24.2823939Z 2022-11-23T02:24:24.2824230Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2824568Z Ran 1 test in 10.682s 2022-11-23T02:24:24.2824732Z 2022-11-23T02:24:24.2824818Z OK 2022-11-23T02:24:24.2824930Z 2022-11-23T02:24:24.2825048Z Generating XML reports... 2022-11-23T02:24:24.2825710Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022335.xml 2022-11-23T02:24:24.2826383Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2827041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2827512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2828140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2828637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2828878Z 2022-11-23T02:24:24.2828964Z Running tests... 2022-11-23T02:24:24.2829403Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2829823Z test_unknown_handler (__main__.RendezvousTest) ... ok (0.476s) 2022-11-23T02:24:24.2830042Z 2022-11-23T02:24:24.2830329Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2830667Z Ran 1 test in 0.476s 2022-11-23T02:24:24.2830828Z 2022-11-23T02:24:24.2830914Z OK 2022-11-23T02:24:24.2831042Z 2022-11-23T02:24:24.2831163Z Generating XML reports... 2022-11-23T02:24:24.2831753Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022349.xml 2022-11-23T02:24:24.2832409Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2833238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2833709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2834336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2834831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2835064Z 2022-11-23T02:24:24.2835168Z Running tests... 2022-11-23T02:24:24.2835594Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2836015Z test_url_with_node_params (__main__.RendezvousTest) ... ok (0.482s) 2022-11-23T02:24:24.2836240Z 2022-11-23T02:24:24.2836526Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2836863Z Ran 1 test in 0.482s 2022-11-23T02:24:24.2837024Z 2022-11-23T02:24:24.2837110Z OK 2022-11-23T02:24:24.2837240Z 2022-11-23T02:24:24.2837361Z Generating XML reports... 2022-11-23T02:24:24.2837966Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022352.xml 2022-11-23T02:24:24.2838657Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2839322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2839801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2840506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2841000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2841233Z 2022-11-23T02:24:24.2841334Z Running tests... 2022-11-23T02:24:24.2841770Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2842537Z test_address_already_in_use (__main__.TCPStoreTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:41599 (errno: 98 - Address already in use). 2022-11-23T02:24:24.2843267Z [W socket.cpp:426] [c10d] The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use). 2022-11-23T02:24:24.2843768Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address. 2022-11-23T02:24:24.2844110Z ok (0.491s) 2022-11-23T02:24:24.2844259Z 2022-11-23T02:24:24.2844605Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2844949Z Ran 1 test in 0.491s 2022-11-23T02:24:24.2845129Z 2022-11-23T02:24:24.2845216Z OK 2022-11-23T02:24:24.2845347Z 2022-11-23T02:24:24.2845451Z Generating XML reports... 2022-11-23T02:24:24.2846053Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022355.xml 2022-11-23T02:24:24.2846707Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2847368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2847843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2848467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2848963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2849196Z 2022-11-23T02:24:24.2849302Z Running tests... 2022-11-23T02:24:24.2849727Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2850136Z test_compare_set (__main__.TCPStoreTest) ... ok (0.507s) 2022-11-23T02:24:24.2850346Z 2022-11-23T02:24:24.2850629Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2850962Z Ran 1 test in 0.507s 2022-11-23T02:24:24.2851123Z 2022-11-23T02:24:24.2851209Z OK 2022-11-23T02:24:24.2851337Z 2022-11-23T02:24:24.2851456Z Generating XML reports... 2022-11-23T02:24:24.2852037Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022359.xml 2022-11-23T02:24:24.2852699Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2853364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2853843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2854468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2854960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2855193Z 2022-11-23T02:24:24.2855293Z Running tests... 2022-11-23T02:24:24.2855716Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2856305Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:24:24.2857165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:24:24.2857579Z ok (0.553s) 2022-11-23T02:24:24.2857726Z 2022-11-23T02:24:24.2858016Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2858356Z Ran 1 test in 0.553s 2022-11-23T02:24:24.2858518Z 2022-11-23T02:24:24.2858604Z OK 2022-11-23T02:24:24.2858737Z 2022-11-23T02:24:24.2858840Z Generating XML reports... 2022-11-23T02:24:24.2859435Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022402.xml 2022-11-23T02:24:24.2860084Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2860748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2861221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2861845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2862348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2862581Z 2022-11-23T02:24:24.2862686Z Running tests... 2022-11-23T02:24:24.2863155Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2863599Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... ok (0.509s) 2022-11-23T02:24:24.2863842Z 2022-11-23T02:24:24.2864128Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2864467Z Ran 1 test in 0.509s 2022-11-23T02:24:24.2864627Z 2022-11-23T02:24:24.2864713Z OK 2022-11-23T02:24:24.2864841Z 2022-11-23T02:24:24.2864959Z Generating XML reports... 2022-11-23T02:24:24.2865541Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022405.xml 2022-11-23T02:24:24.2866198Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2866867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2867339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2867968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2868463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2868694Z 2022-11-23T02:24:24.2868797Z Running tests... 2022-11-23T02:24:24.2869222Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2869669Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... ok (0.491s) 2022-11-23T02:24:24.2869918Z 2022-11-23T02:24:24.2870200Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2870542Z Ran 1 test in 0.491s 2022-11-23T02:24:24.2870704Z 2022-11-23T02:24:24.2870792Z OK 2022-11-23T02:24:24.2870923Z 2022-11-23T02:24:24.2871044Z Generating XML reports... 2022-11-23T02:24:24.2871635Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022409.xml 2022-11-23T02:24:24.2872268Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2872931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2873406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2874027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2874520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2874753Z 2022-11-23T02:24:24.2874856Z Running tests... 2022-11-23T02:24:24.2875291Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2875736Z test_multitenancy (__main__.TCPStoreTest) ... ok (0.488s) 2022-11-23T02:24:24.2875950Z 2022-11-23T02:24:24.2876236Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2876576Z Ran 1 test in 0.488s 2022-11-23T02:24:24.2876737Z 2022-11-23T02:24:24.2876829Z OK 2022-11-23T02:24:24.2876963Z 2022-11-23T02:24:24.2877082Z Generating XML reports... 2022-11-23T02:24:24.2877672Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022412.xml 2022-11-23T02:24:24.2878319Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2878965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2879438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2880058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2880610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2880843Z 2022-11-23T02:24:24.2880945Z Running tests... 2022-11-23T02:24:24.2881383Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2881849Z test_numkeys_delkeys (__main__.TCPStoreTest) ... ok (2.517s) 2022-11-23T02:24:24.2882071Z 2022-11-23T02:24:24.2882342Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2882679Z Ran 1 test in 2.517s 2022-11-23T02:24:24.2882838Z 2022-11-23T02:24:24.2882923Z OK 2022-11-23T02:24:24.2883050Z 2022-11-23T02:24:24.2883169Z Generating XML reports... 2022-11-23T02:24:24.2883760Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022415.xml 2022-11-23T02:24:24.2884408Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:24:24.2885079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:24:24.2885535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:24:24.2886156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:24:24.2886655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:24:24.2886885Z 2022-11-23T02:24:24.2886987Z Running tests... 2022-11-23T02:24:24.2887427Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2887823Z test_set_get (__main__.TCPStoreTest) ... ok (0.490s) 2022-11-23T02:24:24.2888026Z 2022-11-23T02:24:24.2888309Z ---------------------------------------------------------------------- 2022-11-23T02:24:24.2888633Z Ran 1 test in 0.490s 2022-11-23T02:24:24.2888791Z 2022-11-23T02:24:24.2888878Z OK 2022-11-23T02:24:24.2889007Z 2022-11-23T02:24:24.2889128Z Generating XML reports... 2022-11-23T02:24:24.2889722Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022421.xml 2022-11-23T02:24:24.2890055Z 2022-11-23T02:24:24.2890446Z ##[endgroup] 2022-11-23T02:24:24.2891039Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_store_a3op8ur0) 2022-11-23T02:24:24.2891370Z 2022-11-23T02:24:24.2891631Z Running distributed/test_nccl ... [2022-11-23 02:24:24.260688] 2022-11-23T02:24:24.2892397Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:24:24.261539] 2022-11-23T02:24:27.6895835Z 2022-11-23T02:24:27.6897052Z Expand the folded group to see the log file of distributed/test_nccl 2022-11-23T02:24:27.6899335Z ##[group]PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_nccl_j5fsd3r4) 2022-11-23T02:24:27.6902001Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl 2022-11-23T02:24:27.6902676Z 2022-11-23T02:24:27.6902934Z Running tests... 2022-11-23T02:24:27.6904100Z ---------------------------------------------------------------------- 2022-11-23T02:24:27.6906158Z test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6908504Z test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6910811Z test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6913112Z test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6915400Z test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6917915Z test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6920512Z test_collective_errors_cuda (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6922772Z test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6924997Z test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6927308Z test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6929637Z test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:24:27.6931898Z test_unique_id_cuda (__main__.TestNCCLCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:24:27.6932825Z 2022-11-23T02:24:27.6933570Z ---------------------------------------------------------------------- 2022-11-23T02:24:27.6934444Z Ran 12 tests in 0.014s 2022-11-23T02:24:27.6934855Z 2022-11-23T02:24:27.6935114Z OK (skipped=12) 2022-11-23T02:24:27.6935505Z 2022-11-23T02:24:27.6935811Z Generating XML reports... 2022-11-23T02:24:27.6937343Z Generated XML report: test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20221123022426.xml 2022-11-23T02:24:27.6938197Z 2022-11-23T02:24:27.6938975Z ##[endgroup] 2022-11-23T02:24:27.6940467Z FINISHED PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_nccl_j5fsd3r4) 2022-11-23T02:24:27.6941310Z 2022-11-23T02:24:27.6942020Z Running distributed/test_launcher ... [2022-11-23 02:24:27.690239] 2022-11-23T02:24:27.6944046Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:24:27.691085] 2022-11-23T02:25:26.2137321Z 2022-11-23T02:25:26.2140946Z Expand the folded group to see the log file of distributed/test_launcher 2022-11-23T02:25:26.2144248Z ##[group]PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_launcher_or003ci1) 2022-11-23T02:25:26.2146782Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher 2022-11-23T02:25:26.2147499Z 2022-11-23T02:25:26.2149216Z Running tests... 2022-11-23T02:25:26.2150666Z ---------------------------------------------------------------------- 2022-11-23T02:25:26.2152970Z test_launch_user_script (__main__.TestDistributedLaunch) ... /opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py:180: FutureWarning: The module torch.distributed.launch is deprecated 2022-11-23T02:25:26.2154458Z and will be removed in future. Use torchrun. 2022-11-23T02:25:26.2155508Z Note that --use_env is set by default in torchrun. 2022-11-23T02:25:26.2156714Z If your script expects `--local_rank` argument to be set, please 2022-11-23T02:25:26.2158017Z change it to read from `os.environ['LOCAL_RANK']` instead. See 2022-11-23T02:25:26.2159373Z https://pytorch.org/docs/stable/distributed.html#launch-utility for 2022-11-23T02:25:26.2160674Z further instructions 2022-11-23T02:25:26.2161084Z 2022-11-23T02:25:26.2161357Z warnings.warn( 2022-11-23T02:25:26.2162066Z WARNING:torch.distributed.run: 2022-11-23T02:25:26.2162770Z ***************************************** 2022-11-23T02:25:26.2164598Z Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 2022-11-23T02:25:26.2165973Z ***************************************** 2022-11-23T02:25:26.2166610Z Success, smoke test 2022-11-23T02:25:26.2167238Z Success, smoke test 2022-11-23T02:25:26.2167857Z Success, smoke test 2022-11-23T02:25:26.2168446Z Success, smoke test 2022-11-23T02:25:26.2169032Z ok (1.576s) 2022-11-23T02:25:26.2170014Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2170901Z Success, smoke test 2022-11-23T02:25:26.2171512Z Success, smoke test 2022-11-23T02:25:26.2172118Z Success, smoke test 2022-11-23T02:25:26.2173181Z test_launch_user_script succeeded - num_retries_left: 50 2022-11-23T02:25:26.2173936Z ok (1.068s) 2022-11-23T02:25:26.2174820Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2175700Z Success, smoke test 2022-11-23T02:25:26.2176311Z Success, smoke test 2022-11-23T02:25:26.2176923Z Success, smoke test 2022-11-23T02:25:26.2177948Z test_launch_user_script succeeded - num_retries_left: 49 2022-11-23T02:25:26.2178677Z ok (1.068s) 2022-11-23T02:25:26.2179557Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2180417Z Success, smoke test 2022-11-23T02:25:26.2181025Z Success, smoke test 2022-11-23T02:25:26.2181627Z Success, smoke test 2022-11-23T02:25:26.2182615Z test_launch_user_script succeeded - num_retries_left: 48 2022-11-23T02:25:26.2183365Z ok (1.069s) 2022-11-23T02:25:26.2184244Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2185117Z Success, smoke test 2022-11-23T02:25:26.2185750Z Success, smoke test 2022-11-23T02:25:26.2186335Z Success, smoke test 2022-11-23T02:25:26.2187349Z test_launch_user_script succeeded - num_retries_left: 47 2022-11-23T02:25:26.2188097Z ok (1.074s) 2022-11-23T02:25:26.2188968Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2189844Z Success, smoke test 2022-11-23T02:25:26.2190449Z Success, smoke test 2022-11-23T02:25:26.2191028Z Success, smoke test 2022-11-23T02:25:26.2192044Z test_launch_user_script succeeded - num_retries_left: 46 2022-11-23T02:25:26.2192783Z ok (1.083s) 2022-11-23T02:25:26.2193659Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2194524Z Success, smoke test 2022-11-23T02:25:26.2195112Z Success, smoke test 2022-11-23T02:25:26.2195722Z Success, smoke test 2022-11-23T02:25:26.2196740Z test_launch_user_script succeeded - num_retries_left: 45 2022-11-23T02:25:26.2197485Z ok (1.072s) 2022-11-23T02:25:26.2198538Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2199414Z Success, smoke test 2022-11-23T02:25:26.2200004Z Success, smoke test 2022-11-23T02:25:26.2200729Z Success, smoke test 2022-11-23T02:25:26.2201754Z test_launch_user_script succeeded - num_retries_left: 44 2022-11-23T02:25:26.2202502Z ok (1.072s) 2022-11-23T02:25:26.2203386Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2204257Z Success, smoke test 2022-11-23T02:25:26.2204843Z Success, smoke test 2022-11-23T02:25:26.2205450Z Success, smoke test 2022-11-23T02:25:26.2206469Z test_launch_user_script succeeded - num_retries_left: 43 2022-11-23T02:25:26.2207211Z ok (1.071s) 2022-11-23T02:25:26.2208083Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2208924Z Success, smoke test 2022-11-23T02:25:26.2209533Z Success, smoke test 2022-11-23T02:25:26.2210133Z Success, smoke test 2022-11-23T02:25:26.2211150Z test_launch_user_script succeeded - num_retries_left: 42 2022-11-23T02:25:26.2211884Z ok (1.071s) 2022-11-23T02:25:26.2212748Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2213585Z Success, smoke test 2022-11-23T02:25:26.2214195Z Success, smoke test 2022-11-23T02:25:26.2214939Z Success, smoke test 2022-11-23T02:25:26.2215959Z test_launch_user_script succeeded - num_retries_left: 41 2022-11-23T02:25:26.2216700Z ok (1.071s) 2022-11-23T02:25:26.2217555Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2218419Z Success, smoke test 2022-11-23T02:25:26.2219030Z Success, smoke test 2022-11-23T02:25:26.2219635Z Success, smoke test 2022-11-23T02:25:26.2220663Z test_launch_user_script succeeded - num_retries_left: 40 2022-11-23T02:25:26.2221411Z ok (1.071s) 2022-11-23T02:25:26.2222270Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2223132Z Success, smoke test 2022-11-23T02:25:26.2223745Z Success, smoke test 2022-11-23T02:25:26.2224352Z Success, smoke test 2022-11-23T02:25:26.2225373Z test_launch_user_script succeeded - num_retries_left: 39 2022-11-23T02:25:26.2226109Z ok (1.072s) 2022-11-23T02:25:26.2226964Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2227823Z Success, smoke test 2022-11-23T02:25:26.2228431Z Success, smoke test 2022-11-23T02:25:26.2229030Z Success, smoke test 2022-11-23T02:25:26.2230292Z test_launch_user_script succeeded - num_retries_left: 38 2022-11-23T02:25:26.2231035Z ok (1.071s) 2022-11-23T02:25:26.2231903Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2232779Z Success, smoke test 2022-11-23T02:25:26.2233374Z Success, smoke test 2022-11-23T02:25:26.2233972Z Success, smoke test 2022-11-23T02:25:26.2234990Z test_launch_user_script succeeded - num_retries_left: 37 2022-11-23T02:25:26.2235737Z ok (1.072s) 2022-11-23T02:25:26.2236608Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2237467Z Success, smoke test 2022-11-23T02:25:26.2238058Z Success, smoke test 2022-11-23T02:25:26.2238662Z Success, smoke test 2022-11-23T02:25:26.2239680Z test_launch_user_script succeeded - num_retries_left: 36 2022-11-23T02:25:26.2240550Z ok (1.072s) 2022-11-23T02:25:26.2241415Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2242270Z Success, smoke test 2022-11-23T02:25:26.2242862Z Success, smoke test 2022-11-23T02:25:26.2243467Z Success, smoke test 2022-11-23T02:25:26.2244492Z test_launch_user_script succeeded - num_retries_left: 35 2022-11-23T02:25:26.2245231Z ok (1.070s) 2022-11-23T02:25:26.2246107Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2246952Z Success, smoke test 2022-11-23T02:25:26.2247564Z Success, smoke test 2022-11-23T02:25:26.2248328Z Success, smoke test 2022-11-23T02:25:26.2249349Z test_launch_user_script succeeded - num_retries_left: 34 2022-11-23T02:25:26.2250093Z ok (1.073s) 2022-11-23T02:25:26.2250970Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2251819Z Success, smoke test 2022-11-23T02:25:26.2252445Z Success, smoke test 2022-11-23T02:25:26.2253051Z Success, smoke test 2022-11-23T02:25:26.2254077Z test_launch_user_script succeeded - num_retries_left: 33 2022-11-23T02:25:26.2254823Z ok (1.071s) 2022-11-23T02:25:26.2255678Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2256542Z Success, smoke test 2022-11-23T02:25:26.2257154Z Success, smoke test 2022-11-23T02:25:26.2257756Z Success, smoke test 2022-11-23T02:25:26.2258781Z test_launch_user_script succeeded - num_retries_left: 32 2022-11-23T02:25:26.2259528Z ok (1.071s) 2022-11-23T02:25:26.2260378Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2261245Z Success, smoke test 2022-11-23T02:25:26.2261860Z Success, smoke test 2022-11-23T02:25:26.2262461Z Success, smoke test 2022-11-23T02:25:26.2263472Z test_launch_user_script succeeded - num_retries_left: 31 2022-11-23T02:25:26.2264210Z ok (1.071s) 2022-11-23T02:25:26.2265192Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2266062Z Success, smoke test 2022-11-23T02:25:26.2266673Z Success, smoke test 2022-11-23T02:25:26.2267279Z Success, smoke test 2022-11-23T02:25:26.2268298Z test_launch_user_script succeeded - num_retries_left: 30 2022-11-23T02:25:26.2269028Z ok (1.071s) 2022-11-23T02:25:26.2269895Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2270756Z Success, smoke test 2022-11-23T02:25:26.2271370Z Success, smoke test 2022-11-23T02:25:26.2271973Z Success, smoke test 2022-11-23T02:25:26.2272975Z test_launch_user_script succeeded - num_retries_left: 29 2022-11-23T02:25:26.2273696Z ok (1.071s) 2022-11-23T02:25:26.2274554Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2275417Z Success, smoke test 2022-11-23T02:25:26.2276025Z Success, smoke test 2022-11-23T02:25:26.2276625Z Success, smoke test 2022-11-23T02:25:26.2277617Z test_launch_user_script succeeded - num_retries_left: 28 2022-11-23T02:25:26.2278359Z ok (1.071s) 2022-11-23T02:25:26.2279214Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2280072Z Success, smoke test 2022-11-23T02:25:26.2280795Z Success, smoke test 2022-11-23T02:25:26.2281393Z Success, smoke test 2022-11-23T02:25:26.2282376Z test_launch_user_script succeeded - num_retries_left: 27 2022-11-23T02:25:26.2283126Z ok (1.072s) 2022-11-23T02:25:26.2283998Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2284509Z Success, smoke test 2022-11-23T02:25:26.2284764Z Success, smoke test 2022-11-23T02:25:26.2285009Z Success, smoke test 2022-11-23T02:25:26.2285402Z test_launch_user_script succeeded - num_retries_left: 26 2022-11-23T02:25:26.2285697Z ok (1.072s) 2022-11-23T02:25:26.2286039Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2286382Z Success, smoke test 2022-11-23T02:25:26.2286627Z Success, smoke test 2022-11-23T02:25:26.2286855Z Success, smoke test 2022-11-23T02:25:26.2287258Z test_launch_user_script succeeded - num_retries_left: 25 2022-11-23T02:25:26.2287554Z ok (1.073s) 2022-11-23T02:25:26.2287900Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2288244Z Success, smoke test 2022-11-23T02:25:26.2288489Z Success, smoke test 2022-11-23T02:25:26.2288721Z Success, smoke test 2022-11-23T02:25:26.2289122Z test_launch_user_script succeeded - num_retries_left: 24 2022-11-23T02:25:26.2289420Z ok (1.071s) 2022-11-23T02:25:26.2289833Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2290177Z Success, smoke test 2022-11-23T02:25:26.2290406Z Success, smoke test 2022-11-23T02:25:26.2290649Z Success, smoke test 2022-11-23T02:25:26.2291053Z test_launch_user_script succeeded - num_retries_left: 23 2022-11-23T02:25:26.2291353Z ok (1.072s) 2022-11-23T02:25:26.2291705Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2292051Z Success, smoke test 2022-11-23T02:25:26.2292283Z Success, smoke test 2022-11-23T02:25:26.2292533Z Success, smoke test 2022-11-23T02:25:26.2292937Z test_launch_user_script succeeded - num_retries_left: 22 2022-11-23T02:25:26.2293232Z ok (1.071s) 2022-11-23T02:25:26.2293578Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2293922Z Success, smoke test 2022-11-23T02:25:26.2294154Z Success, smoke test 2022-11-23T02:25:26.2294397Z Success, smoke test 2022-11-23T02:25:26.2294804Z test_launch_user_script succeeded - num_retries_left: 21 2022-11-23T02:25:26.2295100Z ok (1.073s) 2022-11-23T02:25:26.2295445Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2295775Z Success, smoke test 2022-11-23T02:25:26.2296021Z Success, smoke test 2022-11-23T02:25:26.2296324Z Success, smoke test 2022-11-23T02:25:26.2296735Z test_launch_user_script succeeded - num_retries_left: 20 2022-11-23T02:25:26.2297037Z ok (1.073s) 2022-11-23T02:25:26.2297385Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2297714Z Success, smoke test 2022-11-23T02:25:26.2297963Z Success, smoke test 2022-11-23T02:25:26.2298211Z Success, smoke test 2022-11-23T02:25:26.2298614Z test_launch_user_script succeeded - num_retries_left: 19 2022-11-23T02:25:26.2298913Z ok (1.072s) 2022-11-23T02:25:26.2299246Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2299593Z Success, smoke test 2022-11-23T02:25:26.2299845Z Success, smoke test 2022-11-23T02:25:26.2300093Z Success, smoke test 2022-11-23T02:25:26.2300499Z test_launch_user_script succeeded - num_retries_left: 18 2022-11-23T02:25:26.2300801Z ok (1.073s) 2022-11-23T02:25:26.2301138Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2301483Z Success, smoke test 2022-11-23T02:25:26.2301734Z Success, smoke test 2022-11-23T02:25:26.2301981Z Success, smoke test 2022-11-23T02:25:26.2302385Z test_launch_user_script succeeded - num_retries_left: 17 2022-11-23T02:25:26.2302688Z ok (1.072s) 2022-11-23T02:25:26.2303020Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2303365Z Success, smoke test 2022-11-23T02:25:26.2303614Z Success, smoke test 2022-11-23T02:25:26.2303862Z Success, smoke test 2022-11-23T02:25:26.2304261Z test_launch_user_script succeeded - num_retries_left: 16 2022-11-23T02:25:26.2304546Z ok (1.071s) 2022-11-23T02:25:26.2304895Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2305237Z Success, smoke test 2022-11-23T02:25:26.2305482Z Success, smoke test 2022-11-23T02:25:26.2305732Z Success, smoke test 2022-11-23T02:25:26.2306136Z test_launch_user_script succeeded - num_retries_left: 15 2022-11-23T02:25:26.2306421Z ok (1.072s) 2022-11-23T02:25:26.2306768Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2307111Z Success, smoke test 2022-11-23T02:25:26.2307363Z Success, smoke test 2022-11-23T02:25:26.2307608Z Success, smoke test 2022-11-23T02:25:26.2308008Z test_launch_user_script succeeded - num_retries_left: 14 2022-11-23T02:25:26.2308293Z ok (1.070s) 2022-11-23T02:25:26.2308636Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2308981Z Success, smoke test 2022-11-23T02:25:26.2309230Z Success, smoke test 2022-11-23T02:25:26.2309529Z Success, smoke test 2022-11-23T02:25:26.2309914Z test_launch_user_script succeeded - num_retries_left: 13 2022-11-23T02:25:26.2310213Z ok (1.074s) 2022-11-23T02:25:26.2310562Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2310907Z Success, smoke test 2022-11-23T02:25:26.2311160Z Success, smoke test 2022-11-23T02:25:26.2311406Z Success, smoke test 2022-11-23T02:25:26.2311801Z test_launch_user_script succeeded - num_retries_left: 12 2022-11-23T02:25:26.2312098Z ok (1.069s) 2022-11-23T02:25:26.2312446Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2312789Z Success, smoke test 2022-11-23T02:25:26.2313037Z Success, smoke test 2022-11-23T02:25:26.2313270Z Success, smoke test 2022-11-23T02:25:26.2313674Z test_launch_user_script succeeded - num_retries_left: 11 2022-11-23T02:25:26.2313971Z ok (1.071s) 2022-11-23T02:25:26.2314317Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2314661Z Success, smoke test 2022-11-23T02:25:26.2314905Z Success, smoke test 2022-11-23T02:25:26.2315136Z Success, smoke test 2022-11-23T02:25:26.2315538Z test_launch_user_script succeeded - num_retries_left: 10 2022-11-23T02:25:26.2315837Z ok (1.071s) 2022-11-23T02:25:26.2316233Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2316577Z Success, smoke test 2022-11-23T02:25:26.2316820Z Success, smoke test 2022-11-23T02:25:26.2317050Z Success, smoke test 2022-11-23T02:25:26.2317451Z test_launch_user_script succeeded - num_retries_left: 9 2022-11-23T02:25:26.2317746Z ok (1.072s) 2022-11-23T02:25:26.2318090Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2318433Z Success, smoke test 2022-11-23T02:25:26.2318664Z Success, smoke test 2022-11-23T02:25:26.2318907Z Success, smoke test 2022-11-23T02:25:26.2319305Z test_launch_user_script succeeded - num_retries_left: 8 2022-11-23T02:25:26.2319605Z ok (1.071s) 2022-11-23T02:25:26.2319951Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2320353Z Success, smoke test 2022-11-23T02:25:26.2320586Z Success, smoke test 2022-11-23T02:25:26.2320833Z Success, smoke test 2022-11-23T02:25:26.2321239Z test_launch_user_script succeeded - num_retries_left: 7 2022-11-23T02:25:26.2321536Z ok (1.071s) 2022-11-23T02:25:26.2321881Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2322224Z Success, smoke test 2022-11-23T02:25:26.2322458Z Success, smoke test 2022-11-23T02:25:26.2322705Z Success, smoke test 2022-11-23T02:25:26.2323102Z test_launch_user_script succeeded - num_retries_left: 6 2022-11-23T02:25:26.2323401Z ok (1.071s) 2022-11-23T02:25:26.2323742Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2324070Z Success, smoke test 2022-11-23T02:25:26.2324320Z Success, smoke test 2022-11-23T02:25:26.2324565Z Success, smoke test 2022-11-23T02:25:26.2324956Z test_launch_user_script succeeded - num_retries_left: 5 2022-11-23T02:25:26.2325249Z ok (1.072s) 2022-11-23T02:25:26.2325593Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2325922Z Success, smoke test 2022-11-23T02:25:26.2326170Z Success, smoke test 2022-11-23T02:25:26.2326416Z Success, smoke test 2022-11-23T02:25:26.2326811Z test_launch_user_script succeeded - num_retries_left: 4 2022-11-23T02:25:26.2327107Z ok (1.072s) 2022-11-23T02:25:26.2327439Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2327781Z Success, smoke test 2022-11-23T02:25:26.2328026Z Success, smoke test 2022-11-23T02:25:26.2328267Z Success, smoke test 2022-11-23T02:25:26.2328664Z test_launch_user_script succeeded - num_retries_left: 3 2022-11-23T02:25:26.2328960Z ok (1.071s) 2022-11-23T02:25:26.2329359Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2329701Z Success, smoke test 2022-11-23T02:25:26.2329946Z Success, smoke test 2022-11-23T02:25:26.2330196Z Success, smoke test 2022-11-23T02:25:26.2330600Z test_launch_user_script succeeded - num_retries_left: 2 2022-11-23T02:25:26.2330884Z ok (1.072s) 2022-11-23T02:25:26.2331235Z test_launch_user_script (__main__.TestDistributedLaunch) ... Success, smoke test 2022-11-23T02:25:26.2331581Z Success, smoke test 2022-11-23T02:25:26.2331834Z Success, smoke test 2022-11-23T02:25:26.2332079Z Success, smoke test 2022-11-23T02:25:26.2332483Z test_launch_user_script succeeded - num_retries_left: 1 2022-11-23T02:25:26.2332763Z ok (1.072s) 2022-11-23T02:25:26.2333147Z test_launch_user_script succeeded - num_retries_left: 0 2022-11-23T02:25:26.2333362Z 2022-11-23T02:25:26.2333654Z ---------------------------------------------------------------------- 2022-11-23T02:25:26.2334000Z Ran 51 tests in 55.192s 2022-11-23T02:25:26.2334163Z 2022-11-23T02:25:26.2334246Z OK 2022-11-23T02:25:26.2334379Z 2022-11-23T02:25:26.2334496Z Generating XML reports... 2022-11-23T02:25:26.2335133Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20221123022429.xml 2022-11-23T02:25:26.2335500Z 2022-11-23T02:25:26.2335959Z ##[endgroup] 2022-11-23T02:25:26.2336651Z FINISHED PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_launcher_or003ci1) 2022-11-23T02:25:26.2336997Z 2022-11-23T02:25:26.2337287Z Running distributed/test_data_parallel ... [2022-11-23 02:25:26.214647] 2022-11-23T02:25:26.2338100Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:25:26.215497] 2022-11-23T02:25:29.6740666Z 2022-11-23T02:25:29.6741866Z Expand the folded group to see the log file of distributed/test_data_parallel 2022-11-23T02:25:29.6745733Z ##[group]PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_data_parallel_56vy61t4) 2022-11-23T02:25:29.6749153Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel 2022-11-23T02:25:29.6750379Z 2022-11-23T02:25:29.6750994Z Running tests... 2022-11-23T02:25:29.6752791Z ---------------------------------------------------------------------- 2022-11-23T02:25:29.6754968Z test_autocast (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6757258Z test_data_parallel (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6759673Z test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6762490Z test_data_parallel_complex (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6765738Z test_data_parallel_device_args (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6768358Z test_data_parallel_function_deletion (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6770746Z test_data_parallel_lazy_linear (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6772179Z test_data_parallel_model_device (__main__.TestDataParallel) 2022-11-23T02:25:29.6773999Z Test device[0] check at forward time. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:25:29.6776761Z test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6779212Z test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6781580Z test_data_parallel_multiple_input (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:25:29.6783939Z test_data_parallel_nested_input (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6786342Z test_data_parallel_nested_output (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6788826Z test_data_parallel_no_grad (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6791116Z test_data_parallel_rnn (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6793429Z test_data_parallel_small_back (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6795735Z test_data_parallel_sparse (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6797987Z test_gather_cpu (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:25:29.6800465Z test_gather_different_len_dicts (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6802750Z test_gather_gpu (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:25:29.6804997Z test_parallel_apply (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6807305Z test_parallel_apply_autocast (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6809713Z test_parallel_apply_passes_exception (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6812125Z test_parameter_list_dict_replica (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6814379Z test_replicate (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6816648Z test_replicate_buffers (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6818944Z test_save_replica_module (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6821180Z test_scatter_cpu (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:25:29.6823549Z test_scatter_gpu (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:25:29.6825773Z test_strided_grad_layout (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:25:29.6828008Z test_zero_grad (__main__.TestDataParallel) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6830500Z test_data_parallel_module_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6833171Z test_data_parallel_module_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6835951Z test_data_parallel_module_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6838696Z test_data_parallel_module_kwargs_only_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6841533Z test_data_parallel_module_kwargs_only_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6844258Z test_data_parallel_module_kwargs_only_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6847067Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6849917Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6852754Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.005s) 2022-11-23T02:25:29.6855531Z test_data_parallel_module_kwargs_only_empty_list_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6858369Z test_data_parallel_module_kwargs_only_empty_list_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6861169Z test_data_parallel_module_kwargs_only_empty_list_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6864024Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6866915Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6869925Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:25:29.6871105Z 2022-11-23T02:25:29.6871854Z ---------------------------------------------------------------------- 2022-11-23T02:25:29.6872704Z Ran 46 tests in 0.053s 2022-11-23T02:25:29.6873111Z 2022-11-23T02:25:29.6873354Z OK (skipped=46) 2022-11-23T02:25:29.6873733Z 2022-11-23T02:25:29.6874029Z Generating XML reports... 2022-11-23T02:25:29.6875649Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221123022528.xml 2022-11-23T02:25:29.6877903Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20221123022528.xml 2022-11-23T02:25:29.6878953Z 2022-11-23T02:25:29.6879746Z ##[endgroup] 2022-11-23T02:25:29.6881479Z FINISHED PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_data_parallel_56vy61t4) 2022-11-23T02:25:29.6882172Z 2022-11-23T02:25:29.6882479Z Running distributed/test_c10d_spawn_nccl ... [2022-11-23 02:25:29.674902] 2022-11-23T02:25:29.6883390Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:25:29.676076] 2022-11-23T02:27:12.2552420Z 2022-11-23T02:27:12.2554051Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl 2022-11-23T02:27:12.2557180Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_nccl_iyiae73f) 2022-11-23T02:27:12.2558782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9rjzr4c 2022-11-23T02:27:12.2560622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9rjzr4c/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2563216Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]> 2022-11-23T02:27:12.2565477Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:27:12.2566609Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:27:12.2567697Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:27:12.2568826Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:27:12.2569790Z 2022-11-23T02:27:12.2570604Z 2022-11-23T02:27:12.2574306Z , <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]> 2022-11-23T02:27:12.2577896Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:27:12.2578991Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:27:12.2580066Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:27:12.2581804Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:27:12.2582883Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:27:12.2583933Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:27:12.2584940Z test_reduce (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:27:12.2586017Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:27:12.2587193Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:27:12.2589245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2590465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2592071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2593328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2594558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsbnhyrd 2022-11-23T02:27:12.2596009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsbnhyrd/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2597703Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2598631Z 2022-11-23T02:27:12.2598882Z Running tests... 2022-11-23T02:27:12.2600009Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2602307Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2603781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2605383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2606610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2607835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdmxcklv 2022-11-23T02:27:12.2609278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdmxcklv/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2611075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2612266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2613849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2615091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2616313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptikpjl7n 2022-11-23T02:27:12.2617750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptikpjl7n/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2618751Z ok (4.492s) 2022-11-23T02:27:12.2619115Z 2022-11-23T02:27:12.2619858Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2620713Z Ran 1 test in 4.494s 2022-11-23T02:27:12.2621107Z 2022-11-23T02:27:12.2621305Z OK 2022-11-23T02:27:12.2621621Z 2022-11-23T02:27:12.2621923Z Generating XML reports... 2022-11-23T02:27:12.2623661Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123022533.xml 2022-11-23T02:27:12.2625440Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:27:12.2627291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2628480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2630057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2631494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2632738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphya0zl15 2022-11-23T02:27:12.2634182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphya0zl15/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2635835Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2636549Z 2022-11-23T02:27:12.2636792Z Running tests... 2022-11-23T02:27:12.2637902Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2639990Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2641640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2643251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2644500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2645873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0qldxqg 2022-11-23T02:27:12.2647321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0qldxqg/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2649127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2650310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2651883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2653125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2654371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphwememg 2022-11-23T02:27:12.2655813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphwememg/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2656787Z ok (4.151s) 2022-11-23T02:27:12.2657144Z 2022-11-23T02:27:12.2657887Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2658732Z Ran 1 test in 4.152s 2022-11-23T02:27:12.2659127Z 2022-11-23T02:27:12.2659327Z OK 2022-11-23T02:27:12.2659645Z 2022-11-23T02:27:12.2659934Z Generating XML reports... 2022-11-23T02:27:12.2661700Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123022541.xml 2022-11-23T02:27:12.2663476Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:27:12.2665332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2666525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2668094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2669337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2670568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyl4m1c2z 2022-11-23T02:27:12.2672022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyl4m1c2z/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2673673Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2674373Z 2022-11-23T02:27:12.2674595Z Running tests... 2022-11-23T02:27:12.2675695Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2677781Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2679382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2681144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2682390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2683618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyczt5rhl 2022-11-23T02:27:12.2684794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyczt5rhl/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2685506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2685979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2686602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2687101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2687590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7t73dqiw 2022-11-23T02:27:12.2688222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7t73dqiw/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2688620Z ok (4.647s) 2022-11-23T02:27:12.2688750Z 2022-11-23T02:27:12.2689046Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2689388Z Ran 1 test in 4.649s 2022-11-23T02:27:12.2689545Z 2022-11-23T02:27:12.2689627Z OK 2022-11-23T02:27:12.2689758Z 2022-11-23T02:27:12.2689877Z Generating XML reports... 2022-11-23T02:27:12.2690562Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123022549.xml 2022-11-23T02:27:12.2691269Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:27:12.2691999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2692451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2693079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2693571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2694062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31oakjuu 2022-11-23T02:27:12.2694631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31oakjuu/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2695284Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2695566Z 2022-11-23T02:27:12.2695666Z Running tests... 2022-11-23T02:27:12.2696090Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2696908Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2697481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2698103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2698588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2699074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml7mrz39 2022-11-23T02:27:12.2699636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml7mrz39/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2700339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2700856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2701474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2701965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2702447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4x76ekmw 2022-11-23T02:27:12.2703012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4x76ekmw/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2703404Z ok (4.119s) 2022-11-23T02:27:12.2703548Z 2022-11-23T02:27:12.2703837Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2704166Z Ran 1 test in 4.120s 2022-11-23T02:27:12.2704326Z 2022-11-23T02:27:12.2704411Z OK 2022-11-23T02:27:12.2704540Z 2022-11-23T02:27:12.2704660Z Generating XML reports... 2022-11-23T02:27:12.2705360Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123022557.xml 2022-11-23T02:27:12.2706063Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:27:12.2706841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2707311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2707925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2708420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2708910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybc2o9nc 2022-11-23T02:27:12.2709480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybc2o9nc/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2710134Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2710491Z 2022-11-23T02:27:12.2710599Z Running tests... 2022-11-23T02:27:12.2711037Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2711612Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6783 2022-11-23T02:27:12.2712171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6784 2022-11-23T02:27:12.2712825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2713297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2713921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2714419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2714907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9zl7p9w 2022-11-23T02:27:12.2715481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9zl7p9w/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2716172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2716636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2717249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2717736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2718217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftyulwy6 2022-11-23T02:27:12.2718783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftyulwy6/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2719536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:27:12.2720035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:27:12.2720625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:27:12.2721144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:27:12.2721868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2722623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2723040Z ok (4.037s) 2022-11-23T02:27:12.2723187Z 2022-11-23T02:27:12.2723491Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2723825Z Ran 1 test in 4.038s 2022-11-23T02:27:12.2723986Z 2022-11-23T02:27:12.2724069Z OK 2022-11-23T02:27:12.2724199Z 2022-11-23T02:27:12.2724317Z Generating XML reports... 2022-11-23T02:27:12.2725080Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022604.xml 2022-11-23T02:27:12.2725893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2726363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2726984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2727458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2727947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4e7b8d83 2022-11-23T02:27:12.2728511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4e7b8d83/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2729154Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2729433Z 2022-11-23T02:27:12.2729534Z Running tests... 2022-11-23T02:27:12.2729975Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2730554Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6977 2022-11-23T02:27:12.2731132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6978 2022-11-23T02:27:12.2731773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2732239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2732860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2733353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2733837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhnt0y8b 2022-11-23T02:27:12.2734405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhnt0y8b/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2735110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2735561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2736182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2736674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2737158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7oc15sw 2022-11-23T02:27:12.2737783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7oc15sw/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2738321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:27:12.2738839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:27:12.2739356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:27:12.2739851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:27:12.2740567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2741320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2742480Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:27:12.2743072Z warnings.warn( 2022-11-23T02:27:12.2743982Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2923: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead. 2022-11-23T02:27:12.2744561Z warnings.warn( 2022-11-23T02:27:12.2745390Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:27:12.2745960Z warnings.warn( 2022-11-23T02:27:12.2746792Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2923: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead. 2022-11-23T02:27:12.2747372Z warnings.warn( 2022-11-23T02:27:12.2747614Z ok (4.443s) 2022-11-23T02:27:12.2747758Z 2022-11-23T02:27:12.2748052Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2748396Z Ran 1 test in 4.443s 2022-11-23T02:27:12.2748556Z 2022-11-23T02:27:12.2748645Z OK 2022-11-23T02:27:12.2748775Z 2022-11-23T02:27:12.2748894Z Generating XML reports... 2022-11-23T02:27:12.2749588Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022611.xml 2022-11-23T02:27:12.2750408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2750878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2751500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2751998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2752491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps696pwk8 2022-11-23T02:27:12.2753062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps696pwk8/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2753703Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2753975Z 2022-11-23T02:27:12.2754074Z Running tests... 2022-11-23T02:27:12.2754511Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2755083Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7171 2022-11-23T02:27:12.2755656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7172 2022-11-23T02:27:12.2756316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2756846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2757472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2757954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2758446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1s65uvv 2022-11-23T02:27:12.2759016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1s65uvv/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2759722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2760261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2760949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2761449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2761921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lcpdomm 2022-11-23T02:27:12.2762550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lcpdomm/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2763090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:27:12.2763611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:27:12.2764127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:27:12.2764632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:27:12.2765342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2766076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2766491Z ok (4.239s) 2022-11-23T02:27:12.2766639Z 2022-11-23T02:27:12.2766931Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2767282Z Ran 1 test in 4.239s 2022-11-23T02:27:12.2767441Z 2022-11-23T02:27:12.2767525Z OK 2022-11-23T02:27:12.2767653Z 2022-11-23T02:27:12.2767770Z Generating XML reports... 2022-11-23T02:27:12.2768482Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022619.xml 2022-11-23T02:27:12.2769275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2769740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2770364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2770855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2771345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeh4udb8d 2022-11-23T02:27:12.2771919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeh4udb8d/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2772570Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2772848Z 2022-11-23T02:27:12.2772947Z Running tests... 2022-11-23T02:27:12.2773368Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2773940Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7365 2022-11-23T02:27:12.2774517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7366 2022-11-23T02:27:12.2775259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2775727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2776350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2776846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2777318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuuarwfje 2022-11-23T02:27:12.2777886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuuarwfje/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2778601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2779072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2779696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2780188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2780670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mkz_3j5 2022-11-23T02:27:12.2781266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mkz_3j5/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2781793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:27:12.2782310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:27:12.2782822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:27:12.2783321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:27:12.2784035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2784787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2785199Z ok (4.644s) 2022-11-23T02:27:12.2785343Z 2022-11-23T02:27:12.2785620Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2785959Z Ran 1 test in 4.645s 2022-11-23T02:27:12.2786118Z 2022-11-23T02:27:12.2786202Z OK 2022-11-23T02:27:12.2786329Z 2022-11-23T02:27:12.2786449Z Generating XML reports... 2022-11-23T02:27:12.2787152Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022627.xml 2022-11-23T02:27:12.2787955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2788426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2789039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2789533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2790019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lyvewfw 2022-11-23T02:27:12.2790591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lyvewfw/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2791242Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2791521Z 2022-11-23T02:27:12.2791622Z Running tests... 2022-11-23T02:27:12.2792058Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2792605Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7559 2022-11-23T02:27:12.2793179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7560 2022-11-23T02:27:12.2793897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2794369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2794990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2795484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2795969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptn2uasny 2022-11-23T02:27:12.2796537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptn2uasny/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2797222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2797692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2798318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2798808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2799293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5i91hde3 2022-11-23T02:27:12.2799896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5i91hde3/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2800496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:27:12.2801003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:27:12.2801519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:27:12.2802037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:27:12.2802756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2803510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2803922Z ok (4.038s) 2022-11-23T02:27:12.2804067Z 2022-11-23T02:27:12.2804366Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2804693Z Ran 1 test in 4.038s 2022-11-23T02:27:12.2804852Z 2022-11-23T02:27:12.2804935Z OK 2022-11-23T02:27:12.2805066Z 2022-11-23T02:27:12.2805185Z Generating XML reports... 2022-11-23T02:27:12.2805887Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022635.xml 2022-11-23T02:27:12.2806693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2807163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2807787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2808259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2808746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9gmgz6cf 2022-11-23T02:27:12.2809309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9gmgz6cf/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2809964Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2810240Z 2022-11-23T02:27:12.2810341Z Running tests... 2022-11-23T02:27:12.2810784Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2811360Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7753 2022-11-23T02:27:12.2811931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7754 2022-11-23T02:27:12.2812644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2813120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2813752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2814247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2814740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_92bkne 2022-11-23T02:27:12.2815308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_92bkne/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2816016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2816472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2817097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2817592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2818133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpds6y_hc7 2022-11-23T02:27:12.2818699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpds6y_hc7/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2819235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:27:12.2819759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:27:12.2820266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:27:12.2820777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:27:12.2821503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2822255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2822666Z ok (4.237s) 2022-11-23T02:27:12.2822968Z 2022-11-23T02:27:12.2823263Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2823605Z Ran 1 test in 4.238s 2022-11-23T02:27:12.2823769Z 2022-11-23T02:27:12.2823854Z OK 2022-11-23T02:27:12.2823969Z 2022-11-23T02:27:12.2824087Z Generating XML reports... 2022-11-23T02:27:12.2824794Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022642.xml 2022-11-23T02:27:12.2825608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2826086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2826716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2827213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2827703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xei0_z2 2022-11-23T02:27:12.2828252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xei0_z2/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2828904Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2829185Z 2022-11-23T02:27:12.2829288Z Running tests... 2022-11-23T02:27:12.2829728Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2830292Z test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7947 2022-11-23T02:27:12.2830913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7948 2022-11-23T02:27:12.2831576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2832032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2832657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2833152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2833641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwah3h5n 2022-11-23T02:27:12.2834213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwah3h5n/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2834918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2835388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2836020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2836493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2837025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xjmcbzf 2022-11-23T02:27:12.2837588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xjmcbzf/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2838121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:27:12.2838640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:27:12.2839150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:27:12.2839658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:27:12.2840411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2841165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2841575Z ok (4.237s) 2022-11-23T02:27:12.2841725Z 2022-11-23T02:27:12.2842016Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2842357Z Ran 1 test in 4.238s 2022-11-23T02:27:12.2842518Z 2022-11-23T02:27:12.2842603Z OK 2022-11-23T02:27:12.2842733Z 2022-11-23T02:27:12.2842853Z Generating XML reports... 2022-11-23T02:27:12.2843541Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022650.xml 2022-11-23T02:27:12.2844345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2844822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2845447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2845940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2846434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyiow1yd 2022-11-23T02:27:12.2847004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyiow1yd/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2847641Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2847918Z 2022-11-23T02:27:12.2848018Z Running tests... 2022-11-23T02:27:12.2848454Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2849032Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8141 2022-11-23T02:27:12.2849673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8142 2022-11-23T02:27:12.2850328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2850795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2851418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2851896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2852387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45_5bokv 2022-11-23T02:27:12.2852952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45_5bokv/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2853651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2854126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2854747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2855237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2855864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ukqytan 2022-11-23T02:27:12.2856428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ukqytan/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2856966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:27:12.2857482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:27:12.2857994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:27:12.2858504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:27:12.2859227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2859975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2860369Z ok (4.037s) 2022-11-23T02:27:12.2860518Z 2022-11-23T02:27:12.2860806Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2861149Z Ran 1 test in 4.038s 2022-11-23T02:27:12.2861309Z 2022-11-23T02:27:12.2861393Z OK 2022-11-23T02:27:12.2861520Z 2022-11-23T02:27:12.2861639Z Generating XML reports... 2022-11-23T02:27:12.2862339Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022658.xml 2022-11-23T02:27:12.2863144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2863598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2864217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2864707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2865200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7l8aoma8 2022-11-23T02:27:12.2865770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7l8aoma8/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2866423Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:27:12.2866704Z 2022-11-23T02:27:12.2866802Z Running tests... 2022-11-23T02:27:12.2867222Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2867821Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8335 2022-11-23T02:27:12.2868480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8336 2022-11-23T02:27:12.2869135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2869611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2870236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2870729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2871205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8kc3gqu 2022-11-23T02:27:12.2871777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8kc3gqu/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2872489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:27:12.2872963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:27:12.2873586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:27:12.2874118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:27:12.2874606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxy7vn55s 2022-11-23T02:27:12.2875169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxy7vn55s/_remote_module_non_scriptable.py 2022-11-23T02:27:12.2875687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:27:12.2876200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:27:12.2876709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:27:12.2877218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:27:12.2877938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2878690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:27:12.2879096Z ok (3.938s) 2022-11-23T02:27:12.2879240Z 2022-11-23T02:27:12.2879514Z ---------------------------------------------------------------------- 2022-11-23T02:27:12.2879855Z Ran 1 test in 3.939s 2022-11-23T02:27:12.2880012Z 2022-11-23T02:27:12.2880096Z OK 2022-11-23T02:27:12.2880281Z 2022-11-23T02:27:12.2880400Z Generating XML reports... 2022-11-23T02:27:12.2881113Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022705.xml 2022-11-23T02:27:12.2881531Z 2022-11-23T02:27:12.2884222Z ##[endgroup] 2022-11-23T02:27:12.2884913Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_nccl_iyiae73f) 2022-11-23T02:27:12.2885274Z 2022-11-23T02:27:12.2885535Z Running distributed/test_c10d_pypg ... [2022-11-23 02:27:12.256873] 2022-11-23T02:27:12.2900291Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:27:12.257726] 2022-11-23T02:27:15.8187904Z 2022-11-23T02:27:15.8189770Z Expand the folded group to see the log file of distributed/test_c10d_pypg 2022-11-23T02:27:15.8193959Z ##[group]PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_pypg_xr4t9b0u) 2022-11-23T02:27:15.8197465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg 2022-11-23T02:27:15.8198718Z 2022-11-23T02:27:15.8199303Z Running tests... 2022-11-23T02:27:15.8202294Z ---------------------------------------------------------------------- 2022-11-23T02:27:15.8204348Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8208273Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.486s) 2022-11-23T02:27:15.8211123Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8214999Z Dynamic module can be checkpointed multiple times with weight sharing ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8217681Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8220799Z DDP works as expected when layer is checkpointed only once. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8223051Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8226085Z DDP works as expected when layer is checkpointed only once. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8228835Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8232012Z Regardless of reentrant or non-reentrant checkpointing impl, ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8234388Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8237596Z Regardless of reentrant or non-reentrant checkpointing impl, ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8239905Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8243266Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8245646Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8248921Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8251306Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8254537Z Checkpointing should work with static graph in the case of checkpointing ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8256938Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8260107Z With reentrant autograd checkpointing impl, DDP will fail when there are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8262510Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8265727Z With reentrant autograd checkpointing impl, DDP will fail when there are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8268186Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8271367Z Test that checkpointing with weight sharing works. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8274035Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:27:15.8277201Z Test that checkpointing with weight sharing works. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8281083Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8284774Z test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:27:15.8288446Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:27:15.8292295Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8296267Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8300533Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8302909Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8306205Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8308717Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8312018Z Dynamic module can be checkpointed multiple times with weight sharing ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8314572Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8317794Z DDP works as expected when layer is checkpointed only once. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8320295Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8323417Z DDP works as expected when layer is checkpointed only once. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8324327Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8325427Z Regardless of reentrant or non-reentrant checkpointing impl, ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8326289Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8327394Z Regardless of reentrant or non-reentrant checkpointing impl, ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.011s) 2022-11-23T02:27:15.8328202Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8329362Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:27:15.8330261Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8331400Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8332528Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8333776Z Checkpointing should work with static graph in the case of checkpointing ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8334908Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8336353Z With reentrant autograd checkpointing impl, DDP will fail when there are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8337403Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8338443Z With reentrant autograd checkpointing impl, DDP will fail when there are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8339191Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8340436Z Test that checkpointing with weight sharing works. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8341069Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:27:15.8341876Z Test that checkpointing with weight sharing works. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8342814Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8343744Z test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8344701Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8345640Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8346602Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:27:15.8347578Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:27:15.8347989Z 2022-11-23T02:27:15.8348284Z ---------------------------------------------------------------------- 2022-11-23T02:27:15.8348631Z Ran 38 tests in 0.535s 2022-11-23T02:27:15.8348794Z 2022-11-23T02:27:15.8348895Z OK (skipped=38) 2022-11-23T02:27:15.8349052Z 2022-11-23T02:27:15.8349169Z Generating XML reports... 2022-11-23T02:27:15.8349844Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20221123022713.xml 2022-11-23T02:27:15.8350705Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20221123022713.xml 2022-11-23T02:27:15.8351071Z 2022-11-23T02:27:15.8351441Z ##[endgroup] 2022-11-23T02:27:15.8352056Z FINISHED PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_pypg_xr4t9b0u) 2022-11-23T02:27:15.8352401Z 2022-11-23T02:27:15.8352695Z Running distributed/test_c10d_nccl ... [2022-11-23 02:27:15.820086] 2022-11-23T02:27:15.8353594Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:27:15.821202] 2022-11-23T02:42:49.2595346Z 2022-11-23T02:42:49.2596422Z Expand the folded group to see the log file of distributed/test_c10d_nccl 2022-11-23T02:42:49.2600042Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_nccl_4y72wl21) 2022-11-23T02:42:49.2609308Z , <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]> 2022-11-23T02:42:49.2615936Z test_all_reduce_coalesced_nccl (__main__.CommTest) 2022-11-23T02:42:49.2617002Z test_broadcast_coalesced_nccl (__main__.CommTest) 2022-11-23T02:42:49.2617987Z test_nccl_barrier (__main__.CommTest) 2022-11-23T02:42:49.2619107Z test_nccl_barrier_device_ids (__main__.CommTest) 2022-11-23T02:42:49.2620178Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) 2022-11-23T02:42:49.2621243Z test_nccl_barrier_timeout (__main__.CommTest) 2022-11-23T02:42:49.2622272Z test_nccl_barrier_timeout_new_group (__main__.CommTest) 2022-11-23T02:42:49.2623384Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) 2022-11-23T02:42:49.2624520Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) 2022-11-23T02:42:49.2625614Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) 2022-11-23T02:42:49.2626594Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) 2022-11-23T02:42:49.2627682Z test_nncl_rank_membership (__main__.CommTest) 2022-11-23T02:42:49.2628764Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) 2022-11-23T02:42:49.2629909Z test_sequence_num_incremented_nccl_default (__main__.CommTest) 2022-11-23T02:42:49.2631134Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) 2022-11-23T02:42:49.2632239Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) 2022-11-23T02:42:49.2633305Z test_sequence_num_set_nccl_new_group (__main__.CommTest) 2022-11-23T02:42:49.2634332Z test_tensor_dtype_complex (__main__.CommTest) 2022-11-23T02:42:49.2635338Z test_tensor_dtype_mismatch (__main__.CommTest) 2022-11-23T02:42:49.2637946Z , <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]> 2022-11-23T02:42:49.2640785Z test_allgather_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:42:49.2642077Z test_allreduce_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:42:49.2643038Z test_broadcast_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:42:49.2644156Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:42:49.2645710Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) 2022-11-23T02:42:49.2646944Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:42:49.2648048Z test_scatter_work_wait_gpu (__main__.CompilerTest) 2022-11-23T02:42:49.2675425Z , <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_packed_sequence>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]> 2022-11-23T02:42:49.2702301Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2703742Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2705063Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2706465Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2707843Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2709148Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2710450Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2711786Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2713118Z test_channels_last_contig (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2714348Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2715785Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2717251Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2718701Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2720291Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2721284Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2721942Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2722475Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2723055Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2723644Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2724242Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2724842Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2725448Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2726007Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2726524Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2727101Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2727675Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2728281Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2728822Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2729339Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2729844Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2730306Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2730829Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2731387Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2731900Z test_failure_recovery (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2732431Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2732998Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2733565Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2734094Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2734690Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2735284Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2735803Z test_fp16 (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2736290Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2736815Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2737309Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2737806Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2738367Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2738878Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2739405Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2739973Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2740565Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2741163Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2741681Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2742257Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2742798Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2743380Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2743963Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2744552Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2745112Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2745562Z test_no_grad (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2746054Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2746585Z test_pass_default_pg (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2747099Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2747646Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2748179Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2748743Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.2749163Z 2022-11-23T02:42:49.2752536Z , <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]> 2022-11-23T02:42:49.2753965Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) 2022-11-23T02:42:49.2754494Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) 2022-11-23T02:42:49.2754989Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) 2022-11-23T02:42:49.2755439Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) 2022-11-23T02:42:49.2755950Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) 2022-11-23T02:42:49.2756437Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) 2022-11-23T02:42:49.2756924Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) 2022-11-23T02:42:49.2757402Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) 2022-11-23T02:42:49.2757853Z test_nccl_timeout (__main__.NcclErrorHandlingTest) 2022-11-23T02:42:49.2759061Z , <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_reduce_scatter_base>]> 2022-11-23T02:42:49.2760368Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) 2022-11-23T02:42:49.2761018Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) 2022-11-23T02:42:49.2761612Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) 2022-11-23T02:42:49.2762243Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) 2022-11-23T02:42:49.2762894Z ]> 2022-11-23T02:42:49.2763530Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) 2022-11-23T02:42:49.2765810Z , <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_nccl_dist_backend_error>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]> 2022-11-23T02:42:49.2768262Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2768733Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2769182Z test_allgather_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2769619Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2769998Z test_barrier (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2770424Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2770859Z test_empty_tensors (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2771292Z test_gather_checks (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2771735Z test_gather_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2772113Z test_gather_stress (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2772560Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2773006Z test_reduce_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2773464Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2773934Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2774389Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2774827Z test_scatter_checks (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2775213Z test_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2775657Z test_scatter_stress (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2776088Z test_send_recv (__main__.ProcessGroupNCCLTest) 2022-11-23T02:42:49.2776601Z ]> 2022-11-23T02:42:49.2777103Z test_common_errors (__main__.RendezvousEnvTest) 2022-11-23T02:42:49.2777501Z 2022-11-23T02:42:49.2777962Z ]> 2022-11-23T02:42:49.2778466Z test_default_store_timeout_nccl (__main__.TimeoutTest) 2022-11-23T02:42:49.2779774Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2780526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2781072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2781761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2782316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2782585Z 2022-11-23T02:42:49.2782673Z Running tests... 2022-11-23T02:42:49.2783251Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2783866Z test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8645 2022-11-23T02:42:49.2784470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8646 2022-11-23T02:42:49.2785198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2785736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2786425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2786936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2787459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2788191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2788722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2789405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2790008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2790529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2791724Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:42:49.2792411Z warnings.warn( 2022-11-23T02:42:49.2793434Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:42:49.2794165Z warnings.warn( 2022-11-23T02:42:49.2794473Z ok (4.210s) 2022-11-23T02:42:49.2794648Z 2022-11-23T02:42:49.2794982Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2795388Z Ran 1 test in 4.210s 2022-11-23T02:42:49.2795580Z 2022-11-23T02:42:49.2795698Z OK 2022-11-23T02:42:49.2795857Z 2022-11-23T02:42:49.2795961Z Generating XML reports... 2022-11-23T02:42:49.2796624Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022718.xml 2022-11-23T02:42:49.2797337Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2798070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2798602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2799291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2799852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2800122Z 2022-11-23T02:42:49.2800311Z Running tests... 2022-11-23T02:42:49.2800831Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2801434Z test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8837 2022-11-23T02:42:49.2802045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8838 2022-11-23T02:42:49.2802770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2803301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2804056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2804613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2805091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2805845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2806377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2807070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2807619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2808147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2808567Z ok (4.412s) 2022-11-23T02:42:49.2808702Z 2022-11-23T02:42:49.2809025Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2809428Z Ran 1 test in 4.412s 2022-11-23T02:42:49.2809616Z 2022-11-23T02:42:49.2809731Z OK 2022-11-23T02:42:49.2809892Z 2022-11-23T02:42:49.2810043Z Generating XML reports... 2022-11-23T02:42:49.2810771Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022725.xml 2022-11-23T02:42:49.2811491Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2812223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2812711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2813397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2813949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2814213Z 2022-11-23T02:42:49.2814349Z Running tests... 2022-11-23T02:42:49.2814850Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2815425Z test_nccl_barrier (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9029 2022-11-23T02:42:49.2816015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9030 2022-11-23T02:42:49.2816688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2817216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2817906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2818462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2818982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2819715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2820240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2820886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2821434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2821954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2822431Z skip: Need at least 4 CUDA devices (3.421s) 2022-11-23T02:42:49.2822658Z 2022-11-23T02:42:49.2822985Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2823390Z Ran 1 test in 3.421s 2022-11-23T02:42:49.2823580Z 2022-11-23T02:42:49.2823712Z OK (skipped=1) 2022-11-23T02:42:49.2823952Z 2022-11-23T02:42:49.2824056Z Generating XML reports... 2022-11-23T02:42:49.2824705Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022733.xml 2022-11-23T02:42:49.2825414Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2826147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2826683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2827369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2827924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2828191Z 2022-11-23T02:42:49.2828280Z Running tests... 2022-11-23T02:42:49.2828774Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2829369Z test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9206 2022-11-23T02:42:49.2829965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9207 2022-11-23T02:42:49.2830687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2831267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2831955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2832544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2833023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2833604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.2834358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2834896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2835583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2836133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2836652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2837183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.2837963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2838777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2839256Z ok (4.125s) 2022-11-23T02:42:49.2839437Z 2022-11-23T02:42:49.2839760Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2840221Z Ran 1 test in 4.126s 2022-11-23T02:42:49.2840416Z 2022-11-23T02:42:49.2840532Z OK 2022-11-23T02:42:49.2840747Z 2022-11-23T02:42:49.2840855Z Generating XML reports... 2022-11-23T02:42:49.2841514Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022739.xml 2022-11-23T02:42:49.2842230Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2842962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2843496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2844184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2844735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2845071Z 2022-11-23T02:42:49.2845160Z Running tests... 2022-11-23T02:42:49.2845667Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2846301Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9394 2022-11-23T02:42:49.2846932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9395 2022-11-23T02:42:49.2847656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2848191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2848885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2849393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2849918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2850501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.2851247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2851836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2852529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2853087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2853614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2854148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.2854925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2855743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2856287Z ok (3.408s) 2022-11-23T02:42:49.2856464Z 2022-11-23T02:42:49.2856792Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2857194Z Ran 1 test in 3.408s 2022-11-23T02:42:49.2857386Z 2022-11-23T02:42:49.2857501Z OK 2022-11-23T02:42:49.2857614Z 2022-11-23T02:42:49.2857764Z Generating XML reports... 2022-11-23T02:42:49.2858415Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022746.xml 2022-11-23T02:42:49.2859129Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2859860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2860400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2861085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2861636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2861902Z 2022-11-23T02:42:49.2861993Z Running tests... 2022-11-23T02:42:49.2862494Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2863082Z test_nccl_barrier_timeout (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9575 2022-11-23T02:42:49.2863679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9576 2022-11-23T02:42:49.2864554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2865090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2865847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2866363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2866888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2867627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2868158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2868839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2869394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2869921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2870347Z skip: Need at least 4 CUDA devices (3.510s) 2022-11-23T02:42:49.2870581Z 2022-11-23T02:42:49.2870906Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2871314Z Ran 1 test in 3.511s 2022-11-23T02:42:49.2871506Z 2022-11-23T02:42:49.2871641Z OK (skipped=1) 2022-11-23T02:42:49.2871824Z 2022-11-23T02:42:49.2871975Z Generating XML reports... 2022-11-23T02:42:49.2872686Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022753.xml 2022-11-23T02:42:49.2873402Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2874091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2874626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2875320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2875884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2876151Z 2022-11-23T02:42:49.2876284Z Running tests... 2022-11-23T02:42:49.2876789Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2877399Z test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9752 2022-11-23T02:42:49.2878004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9753 2022-11-23T02:42:49.2878672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2879204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2879888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2880505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2881066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2881801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2882343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2882992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2883550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2884082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2884560Z skip: Need at least 4 CUDA devices (3.619s) 2022-11-23T02:42:49.2884788Z 2022-11-23T02:42:49.2885115Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2885519Z Ran 1 test in 3.620s 2022-11-23T02:42:49.2885715Z 2022-11-23T02:42:49.2885913Z OK (skipped=1) 2022-11-23T02:42:49.2886094Z 2022-11-23T02:42:49.2886198Z Generating XML reports... 2022-11-23T02:42:49.2886859Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022759.xml 2022-11-23T02:42:49.2887575Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2888312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2888844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2889532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2890089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2890352Z 2022-11-23T02:42:49.2890440Z Running tests... 2022-11-23T02:42:49.2890942Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2892298Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/68319 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.496s) 2022-11-23T02:42:49.2892965Z 2022-11-23T02:42:49.2893289Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2893694Z Ran 1 test in 0.496s 2022-11-23T02:42:49.2893887Z 2022-11-23T02:42:49.2894021Z OK (skipped=1) 2022-11-23T02:42:49.2894209Z 2022-11-23T02:42:49.2894362Z Generating XML reports... 2022-11-23T02:42:49.2895018Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022805.xml 2022-11-23T02:42:49.2895688Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2896427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2896959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2897648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2898207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2898473Z 2022-11-23T02:42:49.2898608Z Running tests... 2022-11-23T02:42:49.2899110Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2899719Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9987 2022-11-23T02:42:49.2900290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9988 2022-11-23T02:42:49.2901011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2901548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2902234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2902798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2903322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2904061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2904550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2905235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2905789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2906313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2906953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.2907744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2908381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.2909149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2909728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:42:49.2910298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:42:49.2911069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.2911876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.2912349Z ok (4.209s) 2022-11-23T02:42:49.2912525Z 2022-11-23T02:42:49.2912851Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2913304Z Ran 1 test in 4.210s 2022-11-23T02:42:49.2913499Z 2022-11-23T02:42:49.2913569Z OK 2022-11-23T02:42:49.2913732Z 2022-11-23T02:42:49.2913882Z Generating XML reports... 2022-11-23T02:42:49.2914537Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022809.xml 2022-11-23T02:42:49.2915257Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2915988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2916527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2917214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2917723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2917989Z 2022-11-23T02:42:49.2918122Z Running tests... 2022-11-23T02:42:49.2918626Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2919235Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10190 2022-11-23T02:42:49.2919846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10191 2022-11-23T02:42:49.2920642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2921184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2921824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2922388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2922920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2923505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.2924253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2924780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2925463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2926017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2926492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2927133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.2927938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2928724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:42:49.2929507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2930142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:42:49.2930924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.2931789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.2932218Z ok (4.013s) 2022-11-23T02:42:49.2932403Z 2022-11-23T02:42:49.2962344Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2962836Z Ran 1 test in 4.014s 2022-11-23T02:42:49.2963039Z 2022-11-23T02:42:49.2963155Z OK 2022-11-23T02:42:49.2963317Z 2022-11-23T02:42:49.2963475Z Generating XML reports... 2022-11-23T02:42:49.2964392Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022816.xml 2022-11-23T02:42:49.2965130Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2965827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2966352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2967036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2967608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2967872Z 2022-11-23T02:42:49.2968010Z Running tests... 2022-11-23T02:42:49.2968668Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2969281Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10384 2022-11-23T02:42:49.2969833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10385 2022-11-23T02:42:49.2970559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2971090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2971774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2972342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2972837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2973423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.2974178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2974652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2975338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2975902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2976432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2977005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.2977869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2978500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:42:49.2979210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2979832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:42:49.2980595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.2981396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.2981867Z ok (4.114s) 2022-11-23T02:42:49.2982044Z 2022-11-23T02:42:49.2982349Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2982755Z Ran 1 test in 4.115s 2022-11-23T02:42:49.2982948Z 2022-11-23T02:42:49.2983060Z OK 2022-11-23T02:42:49.2983174Z 2022-11-23T02:42:49.2983324Z Generating XML reports... 2022-11-23T02:42:49.2983975Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022823.xml 2022-11-23T02:42:49.2984722Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.2985445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2985985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2986644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2987198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2987416Z 2022-11-23T02:42:49.2987549Z Running tests... 2022-11-23T02:42:49.2988051Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.2988646Z test_nncl_rank_membership (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10578 2022-11-23T02:42:49.2989209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10579 2022-11-23T02:42:49.2989935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2990468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2991134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2991649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2992166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.2992744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.2993498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.2993999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.2994685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.2995230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.2995675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.2996225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.2997002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2997600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:42:49.2998425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.2999030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:42:49.2999739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.3000548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.3000937Z ok (3.615s) 2022-11-23T02:42:49.3001083Z 2022-11-23T02:42:49.3001379Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3001721Z Ran 1 test in 3.616s 2022-11-23T02:42:49.3001883Z 2022-11-23T02:42:49.3001969Z OK 2022-11-23T02:42:49.3002099Z 2022-11-23T02:42:49.3002220Z Generating XML reports... 2022-11-23T02:42:49.3002814Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022830.xml 2022-11-23T02:42:49.3003465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3004180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3004652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3005281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3005773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3006004Z 2022-11-23T02:42:49.3006109Z Running tests... 2022-11-23T02:42:49.3006553Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3007107Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10761 2022-11-23T02:42:49.3007657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10762 2022-11-23T02:42:49.3008316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3008794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3009419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3009913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3010374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3010890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3011572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3012028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3012648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3013144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3013609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3014126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3014842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3015403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:42:49.3016090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3016713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:42:49.3017420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.3018157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.3018567Z ok (3.910s) 2022-11-23T02:42:49.3018712Z 2022-11-23T02:42:49.3019006Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3019349Z Ran 1 test in 3.911s 2022-11-23T02:42:49.3019506Z 2022-11-23T02:42:49.3019584Z OK 2022-11-23T02:42:49.3019697Z 2022-11-23T02:42:49.3019815Z Generating XML reports... 2022-11-23T02:42:49.3020399Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022837.xml 2022-11-23T02:42:49.3021039Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3021704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3022176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3022865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3023347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3023566Z 2022-11-23T02:42:49.3023666Z Running tests... 2022-11-23T02:42:49.3024104Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3024659Z test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10957 2022-11-23T02:42:49.3025212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10958 2022-11-23T02:42:49.3025868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3026338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3026965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3027445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3027901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3028421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3029108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3029576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3030203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3030693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3031129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3031646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3032357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3033109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3033666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:42:49.3034171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:42:49.3034921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.3035654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.3036044Z ok (3.924s) 2022-11-23T02:42:49.3036179Z 2022-11-23T02:42:49.3036471Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3036799Z Ran 1 test in 3.925s 2022-11-23T02:42:49.3036958Z 2022-11-23T02:42:49.3037034Z OK 2022-11-23T02:42:49.3037159Z 2022-11-23T02:42:49.3037276Z Generating XML reports... 2022-11-23T02:42:49.3037852Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022843.xml 2022-11-23T02:42:49.3038493Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3039144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3039608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3040285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3040765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3041041Z 2022-11-23T02:42:49.3041140Z Running tests... 2022-11-23T02:42:49.3041578Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3042133Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11151 2022-11-23T02:42:49.3042677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11152 2022-11-23T02:42:49.3043322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3043787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3044405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3044890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3045353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3046010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3046464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3047065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3047539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3047974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3048368Z skip: Need at least 4 CUDA devices (3.513s) 2022-11-23T02:42:49.3048552Z 2022-11-23T02:42:49.3048831Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3049152Z Ran 1 test in 3.514s 2022-11-23T02:42:49.3049303Z 2022-11-23T02:42:49.3049395Z OK (skipped=1) 2022-11-23T02:42:49.3049531Z 2022-11-23T02:42:49.3049643Z Generating XML reports... 2022-11-23T02:42:49.3050220Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022850.xml 2022-11-23T02:42:49.3050851Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3051512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3051969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3052604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3053146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3053371Z 2022-11-23T02:42:49.3053457Z Running tests... 2022-11-23T02:42:49.3053882Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3054416Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11328 2022-11-23T02:42:49.3054947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11329 2022-11-23T02:42:49.3055589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3056042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3056645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3057108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3057549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3058052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3058758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3059214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3059829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3060309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3060750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3061254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3061963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3062701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3063095Z ok (4.019s) 2022-11-23T02:42:49.3063238Z 2022-11-23T02:42:49.3063524Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3063853Z Ran 1 test in 4.020s 2022-11-23T02:42:49.3064007Z 2022-11-23T02:42:49.3064248Z OK 2022-11-23T02:42:49.3064363Z 2022-11-23T02:42:49.3064478Z Generating XML reports... 2022-11-23T02:42:49.3065057Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022857.xml 2022-11-23T02:42:49.3065695Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3066348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3066814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3067427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3067910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3068136Z 2022-11-23T02:42:49.3068224Z Running tests... 2022-11-23T02:42:49.3068646Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3069178Z test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11516 2022-11-23T02:42:49.3069715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11517 2022-11-23T02:42:49.3070368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3070876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3071491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3071954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3072404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3072914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3073589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3074045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3074666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3075152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3075593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3076104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3076845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3077400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:42:49.3078100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3078650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:42:49.3079356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.3080091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:42:49.3080534Z ok (4.015s) 2022-11-23T02:42:49.3080674Z 2022-11-23T02:42:49.3080959Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3081294Z Ran 1 test in 4.015s 2022-11-23T02:42:49.3081453Z 2022-11-23T02:42:49.3081542Z OK 2022-11-23T02:42:49.3081666Z 2022-11-23T02:42:49.3081781Z Generating XML reports... 2022-11-23T02:42:49.3082368Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022904.xml 2022-11-23T02:42:49.3083018Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3083671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3084134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3084752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3085241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3085468Z 2022-11-23T02:42:49.3085566Z Running tests... 2022-11-23T02:42:49.3085997Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3086515Z test_tensor_dtype_complex (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11708 2022-11-23T02:42:49.3087032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11709 2022-11-23T02:42:49.3087673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3088131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3088743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3089281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3089730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3090236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3090911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3091361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3091978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3092458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3092913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3093422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3094138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3094927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3095323Z ok (4.111s) 2022-11-23T02:42:49.3095463Z 2022-11-23T02:42:49.3095749Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3096077Z Ran 1 test in 4.112s 2022-11-23T02:42:49.3096231Z 2022-11-23T02:42:49.3096315Z OK 2022-11-23T02:42:49.3096436Z 2022-11-23T02:42:49.3096549Z Generating XML reports... 2022-11-23T02:42:49.3097136Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022911.xml 2022-11-23T02:42:49.3097779Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3098433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3098893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3099506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3099991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3100223Z 2022-11-23T02:42:49.3100319Z Running tests... 2022-11-23T02:42:49.3100747Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3101264Z test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11896 2022-11-23T02:42:49.3101785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11897 2022-11-23T02:42:49.3102436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3102898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3103512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3103999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3104451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3104965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3105640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3106086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3106694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3107235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3107687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3108216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3108934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3109672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3110817Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:42:49.3111466Z warnings.warn( 2022-11-23T02:42:49.3112421Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:42:49.3113062Z warnings.warn( 2022-11-23T02:42:49.3114074Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:42:49.3114705Z warnings.warn( 2022-11-23T02:42:49.3115654Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:42:49.3116285Z warnings.warn( 2022-11-23T02:42:49.3116505Z ok (3.809s) 2022-11-23T02:42:49.3116643Z 2022-11-23T02:42:49.3116924Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3117256Z Ran 1 test in 3.809s 2022-11-23T02:42:49.3117407Z 2022-11-23T02:42:49.3117482Z OK 2022-11-23T02:42:49.3117611Z 2022-11-23T02:42:49.3117717Z Generating XML reports... 2022-11-23T02:42:49.3118292Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123022918.xml 2022-11-23T02:42:49.3118935Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3119592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3120059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3120741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3121238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3121465Z 2022-11-23T02:42:49.3121564Z Running tests... 2022-11-23T02:42:49.3121987Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3122523Z test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12081 2022-11-23T02:42:49.3123070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12082 2022-11-23T02:42:49.3123719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3124191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3124804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3125352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3125792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3126445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3126912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3127515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3127994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3128449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3128961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3129471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3130176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3130918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3131998Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3132766Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3133697Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3134470Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3134806Z ok (4.021s) 2022-11-23T02:42:49.3134951Z 2022-11-23T02:42:49.3135242Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3135568Z Ran 1 test in 4.022s 2022-11-23T02:42:49.3135728Z 2022-11-23T02:42:49.3135813Z OK 2022-11-23T02:42:49.3135939Z 2022-11-23T02:42:49.3136057Z Generating XML reports... 2022-11-23T02:42:49.3136666Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123022924.xml 2022-11-23T02:42:49.3137330Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3138000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3138476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3139081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3139573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3139803Z 2022-11-23T02:42:49.3139905Z Running tests... 2022-11-23T02:42:49.3140341Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3140886Z test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12271 2022-11-23T02:42:49.3141432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12272 2022-11-23T02:42:49.3142082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3142538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3143159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3143700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3144156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3144830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3145299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3145916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3146400Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3146841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3147362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3147884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3148588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3149323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3150382Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3151157Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3152089Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3152859Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3153179Z ok (4.018s) 2022-11-23T02:42:49.3153320Z 2022-11-23T02:42:49.3153609Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3153944Z Ran 1 test in 4.019s 2022-11-23T02:42:49.3154106Z 2022-11-23T02:42:49.3154191Z OK 2022-11-23T02:42:49.3154321Z 2022-11-23T02:42:49.3154439Z Generating XML reports... 2022-11-23T02:42:49.3155036Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123022931.xml 2022-11-23T02:42:49.3155700Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3156345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3156807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3157428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3157913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3158141Z 2022-11-23T02:42:49.3158240Z Running tests... 2022-11-23T02:42:49.3158676Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3159209Z test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12461 2022-11-23T02:42:49.3159746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12462 2022-11-23T02:42:49.3160448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3160921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3161543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3162117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3162579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3163250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3163701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3164322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3164806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3165262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3165776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3166314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3167023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3167810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3168817Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3169582Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3170518Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3171285Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3171614Z ok (4.112s) 2022-11-23T02:42:49.3171755Z 2022-11-23T02:42:49.3172044Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3172387Z Ran 1 test in 4.112s 2022-11-23T02:42:49.3172545Z 2022-11-23T02:42:49.3172648Z OK 2022-11-23T02:42:49.3172774Z 2022-11-23T02:42:49.3172878Z Generating XML reports... 2022-11-23T02:42:49.3173476Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123022938.xml 2022-11-23T02:42:49.3174139Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3174797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3175264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3175886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3176369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3176596Z 2022-11-23T02:42:49.3176695Z Running tests... 2022-11-23T02:42:49.3177122Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3177663Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12651 2022-11-23T02:42:49.3178208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12652 2022-11-23T02:42:49.3178858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3179326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3179938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3180481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3180928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3181591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3182052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3182668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3183151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3183603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3184113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3184628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3185335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3186124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3187147Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3187917Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3188846Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3189627Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3190553Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3191318Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3192246Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3193008Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3193923Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3194679Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3195600Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3196366Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3197283Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3198098Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3199014Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3199768Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3200098Z ok (4.210s) 2022-11-23T02:42:49.3200303Z 2022-11-23T02:42:49.3200594Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3200922Z Ran 1 test in 4.211s 2022-11-23T02:42:49.3201079Z 2022-11-23T02:42:49.3201161Z OK 2022-11-23T02:42:49.3201289Z 2022-11-23T02:42:49.3201408Z Generating XML reports... 2022-11-23T02:42:49.3202019Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123022945.xml 2022-11-23T02:42:49.3202690Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3203353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3203829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3204498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3204987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3205215Z 2022-11-23T02:42:49.3205312Z Running tests... 2022-11-23T02:42:49.3205754Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3206307Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12841 2022-11-23T02:42:49.3206861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12842 2022-11-23T02:42:49.3207523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3207992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3208604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3209095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3209555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3210219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3210685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3211306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3211799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3212245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3212757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3213283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3213989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3214732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3215745Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3216571Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3217503Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3218273Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3219196Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3219957Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3220896Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3221661Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3222037Z ok (4.111s) 2022-11-23T02:42:49.3222182Z 2022-11-23T02:42:49.3222474Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3222810Z Ran 1 test in 4.111s 2022-11-23T02:42:49.3222966Z 2022-11-23T02:42:49.3223047Z OK 2022-11-23T02:42:49.3223182Z 2022-11-23T02:42:49.3223301Z Generating XML reports... 2022-11-23T02:42:49.3223891Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123022953.xml 2022-11-23T02:42:49.3224556Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3225216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3225692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3226309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3226802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3227037Z 2022-11-23T02:42:49.3227136Z Running tests... 2022-11-23T02:42:49.3227559Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3228101Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13031 2022-11-23T02:42:49.3228654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13032 2022-11-23T02:42:49.3229309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3229772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3230394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3230886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3231333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3231994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3232457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3233075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3233563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3234010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3234567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3235075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3235791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3236539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3237551Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3238319Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3239256Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3240081Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3241069Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3241836Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3242760Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3243529Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3243847Z ok (4.013s) 2022-11-23T02:42:49.3243993Z 2022-11-23T02:42:49.3244284Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3244626Z Ran 1 test in 4.013s 2022-11-23T02:42:49.3244786Z 2022-11-23T02:42:49.3244870Z OK 2022-11-23T02:42:49.3244996Z 2022-11-23T02:42:49.3245113Z Generating XML reports... 2022-11-23T02:42:49.3245715Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023000.xml 2022-11-23T02:42:49.3246378Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3247022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3247491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3248118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3248607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3248838Z 2022-11-23T02:42:49.3248936Z Running tests... 2022-11-23T02:42:49.3249368Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3249896Z test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13221 2022-11-23T02:42:49.3250423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13222 2022-11-23T02:42:49.3251070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3251531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3252145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3252687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3253142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3253808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3254276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3254875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3255356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3255811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3256327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3256853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3257566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3258356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3259360Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3260134Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3261063Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T02:42:49.3261833Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T02:42:49.3262164Z ok (4.212s) 2022-11-23T02:42:49.3262306Z 2022-11-23T02:42:49.3262592Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3262929Z Ran 1 test in 4.212s 2022-11-23T02:42:49.3263085Z 2022-11-23T02:42:49.3263163Z OK 2022-11-23T02:42:49.3263288Z 2022-11-23T02:42:49.3263401Z Generating XML reports... 2022-11-23T02:42:49.3263984Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123023007.xml 2022-11-23T02:42:49.3264646Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3265309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3265779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3266564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3267052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3267280Z 2022-11-23T02:42:49.3267379Z Running tests... 2022-11-23T02:42:49.3267799Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3268393Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13411 2022-11-23T02:42:49.3268984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13412 2022-11-23T02:42:49.3269639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3270102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3270717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3271252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3271695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3272358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3272820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3273436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3273920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3274372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3274893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglbfx880 2022-11-23T02:42:49.3275452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglbfx880/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3276000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcuyj59x2 2022-11-23T02:42:49.3276605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcuyj59x2/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3277145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3277652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3278011Z ok (6.622s) 2022-11-23T02:42:49.3278152Z 2022-11-23T02:42:49.3278449Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3278775Z Ran 1 test in 6.622s 2022-11-23T02:42:49.3278930Z 2022-11-23T02:42:49.3279009Z OK 2022-11-23T02:42:49.3279133Z 2022-11-23T02:42:49.3279250Z Generating XML reports... 2022-11-23T02:42:49.3279935Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023014.xml 2022-11-23T02:42:49.3280702Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3281372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3281836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3282437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3282920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3283152Z 2022-11-23T02:42:49.3283253Z Running tests... 2022-11-23T02:42:49.3283690Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3284303Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13609 2022-11-23T02:42:49.3284922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13610 2022-11-23T02:42:49.3285579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3286054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3286665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3287155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3287612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3288272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3288743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3289426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3289918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3290367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3290896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5lootzp 2022-11-23T02:42:49.3291552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5lootzp/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3292317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnruwuidd 2022-11-23T02:42:49.3292974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnruwuidd/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3293600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3294183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3294628Z ok (6.318s) 2022-11-23T02:42:49.3294863Z 2022-11-23T02:42:49.3295228Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3295673Z Ran 1 test in 6.319s 2022-11-23T02:42:49.3295949Z 2022-11-23T02:42:49.3296077Z OK 2022-11-23T02:42:49.3296282Z 2022-11-23T02:42:49.3296442Z Generating XML reports... 2022-11-23T02:42:49.3297153Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023023.xml 2022-11-23T02:42:49.3297954Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3298719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3299268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3300030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3300618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3300915Z 2022-11-23T02:42:49.3301084Z Running tests... 2022-11-23T02:42:49.3301639Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3302254Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13807 2022-11-23T02:42:49.3302959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13808 2022-11-23T02:42:49.3303694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3304249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3304952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3305568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3306222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3306977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3307474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3308196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3308791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3309328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3310099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w9it8aa 2022-11-23T02:42:49.3310870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w9it8aa/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3311512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpog7rpu7d 2022-11-23T02:42:49.3312168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpog7rpu7d/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3312826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3313576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3314124Z ok (6.224s) 2022-11-23T02:42:49.3314306Z 2022-11-23T02:42:49.3314652Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3315174Z Ran 1 test in 6.224s 2022-11-23T02:42:49.3315410Z 2022-11-23T02:42:49.3315535Z OK 2022-11-23T02:42:49.3315765Z 2022-11-23T02:42:49.3315926Z Generating XML reports... 2022-11-23T02:42:49.3316828Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023032.xml 2022-11-23T02:42:49.3317679Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3318526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3319085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3319811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3334766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3335073Z 2022-11-23T02:42:49.3335182Z Running tests... 2022-11-23T02:42:49.3335698Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3336329Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14005 2022-11-23T02:42:49.3336950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14006 2022-11-23T02:42:49.3337649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3338131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3338764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3339265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3339726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3340392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3340861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3341490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3341966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3342426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3342966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajjrp3sy 2022-11-23T02:42:49.3343555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajjrp3sy/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3344166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtz_es8c 2022-11-23T02:42:49.3344748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtz_es8c/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3345320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3345985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3346346Z ok (6.115s) 2022-11-23T02:42:49.3346491Z 2022-11-23T02:42:49.3346792Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3347118Z Ran 1 test in 6.116s 2022-11-23T02:42:49.3347277Z 2022-11-23T02:42:49.3347364Z OK 2022-11-23T02:42:49.3347495Z 2022-11-23T02:42:49.3347614Z Generating XML reports... 2022-11-23T02:42:49.3348294Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023042.xml 2022-11-23T02:42:49.3349017Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3349687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3350162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3350766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3351263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3351495Z 2022-11-23T02:42:49.3351599Z Running tests... 2022-11-23T02:42:49.3352030Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3352688Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14203 2022-11-23T02:42:49.3353277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14204 2022-11-23T02:42:49.3353932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3354388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3355012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3355503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3355962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3356820Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3357813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3358277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3358898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3359397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3359846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3360762Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3361616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5k1vnuo 2022-11-23T02:42:49.3362194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5k1vnuo/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3362753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0kedjkx 2022-11-23T02:42:49.3363318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0kedjkx/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3363784Z ok (4.113s) 2022-11-23T02:42:49.3363932Z 2022-11-23T02:42:49.3364231Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3364557Z Ran 1 test in 4.114s 2022-11-23T02:42:49.3364715Z 2022-11-23T02:42:49.3364800Z OK 2022-11-23T02:42:49.3364929Z 2022-11-23T02:42:49.3365048Z Generating XML reports... 2022-11-23T02:42:49.3365719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023051.xml 2022-11-23T02:42:49.3366450Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3367130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3367635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3368241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3368741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3368974Z 2022-11-23T02:42:49.3369075Z Running tests... 2022-11-23T02:42:49.3369513Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3370161Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14399 2022-11-23T02:42:49.3370752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14400 2022-11-23T02:42:49.3371427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3371916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3372711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3373234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3373682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3374538Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3375695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3376161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3376778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3377270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3377736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3378591Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3379434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaevl8z_k 2022-11-23T02:42:49.3380015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaevl8z_k/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3380570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4vwg74e 2022-11-23T02:42:49.3381134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4vwg74e/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3381619Z ok (4.615s) 2022-11-23T02:42:49.3381768Z 2022-11-23T02:42:49.3382064Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3382407Z Ran 1 test in 4.616s 2022-11-23T02:42:49.3382570Z 2022-11-23T02:42:49.3382657Z OK 2022-11-23T02:42:49.3382787Z 2022-11-23T02:42:49.3382908Z Generating XML reports... 2022-11-23T02:42:49.3383568Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023058.xml 2022-11-23T02:42:49.3384286Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3384954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3385427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3386055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3386557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3386789Z 2022-11-23T02:42:49.3386891Z Running tests... 2022-11-23T02:42:49.3387314Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3387957Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14595 2022-11-23T02:42:49.3388545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14596 2022-11-23T02:42:49.3389207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3389679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3390319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3390807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3391253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3391912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3392383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3393037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3393548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3394018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3394559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbwigtw3 2022-11-23T02:42:49.3395146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbwigtw3/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3395709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxp6u8911 2022-11-23T02:42:49.3396262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxp6u8911/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3396635Z ok (4.121s) 2022-11-23T02:42:49.3396779Z 2022-11-23T02:42:49.3397076Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3397411Z Ran 1 test in 4.122s 2022-11-23T02:42:49.3397568Z 2022-11-23T02:42:49.3397650Z OK 2022-11-23T02:42:49.3397775Z 2022-11-23T02:42:49.3397889Z Generating XML reports... 2022-11-23T02:42:49.3398560Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023105.xml 2022-11-23T02:42:49.3399278Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3399932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3400523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3401157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3401648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3401883Z 2022-11-23T02:42:49.3401985Z Running tests... 2022-11-23T02:42:49.3402424Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3403030Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14791 2022-11-23T02:42:49.3403622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14792 2022-11-23T02:42:49.3404278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3404751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3405368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3405860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3406369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3407020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3407476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3408093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3408579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3409033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3409562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd97hq1y_ 2022-11-23T02:42:49.3410130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd97hq1y_/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3410694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_g3quk3 2022-11-23T02:42:49.3411249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_g3quk3/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3411630Z ok (4.013s) 2022-11-23T02:42:49.3411771Z 2022-11-23T02:42:49.3412062Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3412401Z Ran 1 test in 4.013s 2022-11-23T02:42:49.3412569Z 2022-11-23T02:42:49.3412667Z OK 2022-11-23T02:42:49.3412797Z 2022-11-23T02:42:49.3412929Z Generating XML reports... 2022-11-23T02:42:49.3413631Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023112.xml 2022-11-23T02:42:49.3414382Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3415057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3415523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3416141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3416629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3416864Z 2022-11-23T02:42:49.3416987Z Running tests... 2022-11-23T02:42:49.3417447Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3418030Z test_channels_last_contig (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14987 2022-11-23T02:42:49.3418672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14988 2022-11-23T02:42:49.3419331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3419782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3420421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3420912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3421369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3422033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3422498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3423110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3423588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3424046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3424401Z ok (4.015s) 2022-11-23T02:42:49.3424544Z 2022-11-23T02:42:49.3424879Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3425215Z Ran 1 test in 4.016s 2022-11-23T02:42:49.3425369Z 2022-11-23T02:42:49.3425454Z OK 2022-11-23T02:42:49.3425576Z 2022-11-23T02:42:49.3425690Z Generating XML reports... 2022-11-23T02:42:49.3426396Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023119.xml 2022-11-23T02:42:49.3427111Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3427776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3428250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3428877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3429369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3429606Z 2022-11-23T02:42:49.3429708Z Running tests... 2022-11-23T02:42:49.3430139Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3430589Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3431362Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15175 2022-11-23T02:42:49.3431950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15176 2022-11-23T02:42:49.3432596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3433066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3433705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3434199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3434663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3435313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3435803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3436442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3436937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3437476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3437999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjj_ng3_w 2022-11-23T02:42:49.3438563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjj_ng3_w/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3439115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5a3b69t 2022-11-23T02:42:49.3439683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5a3b69t/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3440080Z ok (6.318s) 2022-11-23T02:42:49.3440286Z 2022-11-23T02:42:49.3440584Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3440925Z Ran 1 test in 6.318s 2022-11-23T02:42:49.3441083Z 2022-11-23T02:42:49.3441166Z OK 2022-11-23T02:42:49.3441294Z 2022-11-23T02:42:49.3441410Z Generating XML reports... 2022-11-23T02:42:49.3442076Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023126.xml 2022-11-23T02:42:49.3442804Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3443532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3444017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3444769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3445265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3445499Z 2022-11-23T02:42:49.3445599Z Running tests... 2022-11-23T02:42:49.3446025Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3446511Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3447137Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15373 2022-11-23T02:42:49.3447730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15374 2022-11-23T02:42:49.3448395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3448862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3449483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3449976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3450417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3451081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3451547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3452162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3452650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3453107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3453657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54yalkvq 2022-11-23T02:42:49.3454207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54yalkvq/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3454769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13avx8vx 2022-11-23T02:42:49.3455332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13avx8vx/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3455725Z ok (6.333s) 2022-11-23T02:42:49.3455957Z 2022-11-23T02:42:49.3456253Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3456593Z Ran 1 test in 6.333s 2022-11-23T02:42:49.3456752Z 2022-11-23T02:42:49.3456836Z OK 2022-11-23T02:42:49.3456950Z 2022-11-23T02:42:49.3457070Z Generating XML reports... 2022-11-23T02:42:49.3457748Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023135.xml 2022-11-23T02:42:49.3458467Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3459143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3459614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3460239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3460734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3460965Z 2022-11-23T02:42:49.3461053Z Running tests... 2022-11-23T02:42:49.3461494Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3461979Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3462635Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15571 2022-11-23T02:42:49.3463203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15572 2022-11-23T02:42:49.3463859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3464335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3464942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3465438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3465897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3466553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3467028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3467641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3468132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3468590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3469115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72ev9w15 2022-11-23T02:42:49.3469670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72ev9w15/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3470232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9kpwfm4 2022-11-23T02:42:49.3470798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9kpwfm4/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3471338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3471840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3472347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3472855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3474154Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:42:49.3474988Z warnings.warn( 2022-11-23T02:42:49.3475367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3476655Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:42:49.3477422Z warnings.warn( 2022-11-23T02:42:49.3477813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3478322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3478830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3479333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3479829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3480416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3480920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3481275Z ok (6.224s) 2022-11-23T02:42:49.3481422Z 2022-11-23T02:42:49.3481723Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3482069Z Ran 1 test in 6.225s 2022-11-23T02:42:49.3482230Z 2022-11-23T02:42:49.3482316Z OK 2022-11-23T02:42:49.3482451Z 2022-11-23T02:42:49.3482574Z Generating XML reports... 2022-11-23T02:42:49.3483241Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023145.xml 2022-11-23T02:42:49.3483971Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3484644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3485126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3485757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3486258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3486490Z 2022-11-23T02:42:49.3486593Z Running tests... 2022-11-23T02:42:49.3487021Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3487507Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3488105Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15769 2022-11-23T02:42:49.3488672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15770 2022-11-23T02:42:49.3489337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3489816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3490449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3490932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3491395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3492069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3492595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3493222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3493721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3494187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3494724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcju8h82 2022-11-23T02:42:49.3495316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcju8h82/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3495881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo162_qcx 2022-11-23T02:42:49.3496448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo162_qcx/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3496991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3497510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3498013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3498578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3499869Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:42:49.3500639Z warnings.warn( 2022-11-23T02:42:49.3501030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3502307Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:42:49.3503082Z warnings.warn( 2022-11-23T02:42:49.3503479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3503982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3504487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3504988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3505475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3505964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3506459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3506813Z ok (6.635s) 2022-11-23T02:42:49.3506958Z 2022-11-23T02:42:49.3507256Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3507603Z Ran 1 test in 6.636s 2022-11-23T02:42:49.3507765Z 2022-11-23T02:42:49.3507849Z OK 2022-11-23T02:42:49.3508134Z 2022-11-23T02:42:49.3508239Z Generating XML reports... 2022-11-23T02:42:49.3508918Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023154.xml 2022-11-23T02:42:49.3509646Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3510313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3510785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3511475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3511971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3512208Z 2022-11-23T02:42:49.3512295Z Running tests... 2022-11-23T02:42:49.3512734Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3513242Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3514032Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15967 2022-11-23T02:42:49.3514604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15968 2022-11-23T02:42:49.3515260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3515739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3516364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3516845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3517394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3518071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3518540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3519168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3519666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3520187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3520712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xzdm6i0 2022-11-23T02:42:49.3521291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xzdm6i0/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3521858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemaf4zf7 2022-11-23T02:42:49.3522432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemaf4zf7/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3522981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3523492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3524000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3524493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3524861Z ok (6.216s) 2022-11-23T02:42:49.3525007Z 2022-11-23T02:42:49.3525305Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3525650Z Ran 1 test in 6.217s 2022-11-23T02:42:49.3525807Z 2022-11-23T02:42:49.3525895Z OK 2022-11-23T02:42:49.3526021Z 2022-11-23T02:42:49.3526143Z Generating XML reports... 2022-11-23T02:42:49.3526826Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023203.xml 2022-11-23T02:42:49.3527536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3528201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3528669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3529294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3529850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3530085Z 2022-11-23T02:42:49.3530187Z Running tests... 2022-11-23T02:42:49.3530631Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3531128Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3531908Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16165 2022-11-23T02:42:49.3532481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16166 2022-11-23T02:42:49.3533143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3533615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3534234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3534731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3535194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3535902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3536367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3536985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3537466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3537923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3538454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzgspl2e4 2022-11-23T02:42:49.3539029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzgspl2e4/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3539577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fds6v_l 2022-11-23T02:42:49.3540143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fds6v_l/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3540686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3541196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3541695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3542194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3542556Z ok (6.621s) 2022-11-23T02:42:49.3542702Z 2022-11-23T02:42:49.3542986Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3543330Z Ran 1 test in 6.621s 2022-11-23T02:42:49.3543489Z 2022-11-23T02:42:49.3543574Z OK 2022-11-23T02:42:49.3543701Z 2022-11-23T02:42:49.3543819Z Generating XML reports... 2022-11-23T02:42:49.3544498Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023213.xml 2022-11-23T02:42:49.3545226Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3545891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3546344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3546971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3547466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3547700Z 2022-11-23T02:42:49.3547801Z Running tests... 2022-11-23T02:42:49.3548314Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3548806Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3549625Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16363 2022-11-23T02:42:49.3550201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16364 2022-11-23T02:42:49.3550856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3551324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3551949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3552441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3552902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3553570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3554045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3554717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3555218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3555681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3556218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1susx7_s 2022-11-23T02:42:49.3556797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1susx7_s/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3557366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2gofibg 2022-11-23T02:42:49.3557939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2gofibg/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3558471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3558991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3560108Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:42:49.3561821Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:42:49.3562886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3563377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3563739Z ok (6.526s) 2022-11-23T02:42:49.3563888Z 2022-11-23T02:42:49.3564195Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3564537Z Ran 1 test in 6.526s 2022-11-23T02:42:49.3564698Z 2022-11-23T02:42:49.3564853Z OK 2022-11-23T02:42:49.3564981Z 2022-11-23T02:42:49.3565103Z Generating XML reports... 2022-11-23T02:42:49.3565770Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023222.xml 2022-11-23T02:42:49.3566503Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3567180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3567664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3568298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3568794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3569032Z 2022-11-23T02:42:49.3569137Z Running tests... 2022-11-23T02:42:49.3569572Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3570061Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3570851Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16561 2022-11-23T02:42:49.3571493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16562 2022-11-23T02:42:49.3572154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3572624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3573254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3573754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3574199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3574871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3575342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3575973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3576473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3576935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3577469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt971_nkn 2022-11-23T02:42:49.3578046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt971_nkn/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3578603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sujfkuy 2022-11-23T02:42:49.3579171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sujfkuy/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3579567Z ok (6.334s) 2022-11-23T02:42:49.3579713Z 2022-11-23T02:42:49.3580008Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3580353Z Ran 1 test in 6.334s 2022-11-23T02:42:49.3580514Z 2022-11-23T02:42:49.3580600Z OK 2022-11-23T02:42:49.3580735Z 2022-11-23T02:42:49.3580842Z Generating XML reports... 2022-11-23T02:42:49.3581520Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023232.xml 2022-11-23T02:42:49.3582247Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3582918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3583387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3584013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3584583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3584816Z 2022-11-23T02:42:49.3584902Z Running tests... 2022-11-23T02:42:49.3585345Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3585832Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3586450Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16759 2022-11-23T02:42:49.3587023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16760 2022-11-23T02:42:49.3587694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3588169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3588796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3589283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3589745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3590470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3590941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3591568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3592062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3592521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3593031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3523_9_ 2022-11-23T02:42:49.3593600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3523_9_/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3594152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9k_uxvj 2022-11-23T02:42:49.3594717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9k_uxvj/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3595255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3595765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3596270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3596757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3597117Z ok (6.518s) 2022-11-23T02:42:49.3597258Z 2022-11-23T02:42:49.3597555Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3597665Z Ran 1 test in 6.519s 2022-11-23T02:42:49.3597673Z 2022-11-23T02:42:49.3597758Z OK 2022-11-23T02:42:49.3597764Z 2022-11-23T02:42:49.3597881Z Generating XML reports... 2022-11-23T02:42:49.3598392Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023241.xml 2022-11-23T02:42:49.3598731Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3599135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3599309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3599721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3599916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3599987Z 2022-11-23T02:42:49.3600090Z Running tests... 2022-11-23T02:42:49.3600445Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3600713Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3601058Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16957 2022-11-23T02:42:49.3601286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16958 2022-11-23T02:42:49.3601674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3601854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3602264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3602460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3602698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3603091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3603328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3603751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3603941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3604180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3604444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9z4ooocw 2022-11-23T02:42:49.3604723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9z4ooocw/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3604985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxcvvxfi 2022-11-23T02:42:49.3605263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxcvvxfi/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3606297Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:42:49.3607129Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:42:49.3608162Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:42:49.3608273Z warnings.warn( 2022-11-23T02:42:49.3608518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3609535Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:42:49.3609698Z warnings.warn( 2022-11-23T02:42:49.3609946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3610180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3610415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3610509Z ok (6.631s) 2022-11-23T02:42:49.3610517Z 2022-11-23T02:42:49.3610809Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3610915Z Ran 1 test in 6.632s 2022-11-23T02:42:49.3610922Z 2022-11-23T02:42:49.3611006Z OK 2022-11-23T02:42:49.3611016Z 2022-11-23T02:42:49.3611136Z Generating XML reports... 2022-11-23T02:42:49.3611645Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023250.xml 2022-11-23T02:42:49.3611984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3612435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3612618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3613031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3613227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3613234Z 2022-11-23T02:42:49.3613320Z Running tests... 2022-11-23T02:42:49.3613611Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3613885Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3614224Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17155 2022-11-23T02:42:49.3614454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17156 2022-11-23T02:42:49.3614855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3615032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3615442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3615635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3615870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3616267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3616446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3616853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3617047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3617284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3617554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe6fb2dz 2022-11-23T02:42:49.3617836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe6fb2dz/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3618093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pj8ivrl 2022-11-23T02:42:49.3618370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pj8ivrl/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3619473Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:42:49.3619582Z warnings.warn( 2022-11-23T02:42:49.3619826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3620843Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:42:49.3620952Z warnings.warn( 2022-11-23T02:42:49.3621194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3621430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3621707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3621802Z ok (6.421s) 2022-11-23T02:42:49.3621810Z 2022-11-23T02:42:49.3622106Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3622216Z Ran 1 test in 6.422s 2022-11-23T02:42:49.3622224Z 2022-11-23T02:42:49.3622310Z OK 2022-11-23T02:42:49.3622316Z 2022-11-23T02:42:49.3622436Z Generating XML reports... 2022-11-23T02:42:49.3622948Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023300.xml 2022-11-23T02:42:49.3623285Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3623692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3623869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3624290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3624485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3624492Z 2022-11-23T02:42:49.3624593Z Running tests... 2022-11-23T02:42:49.3624880Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3625148Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3625456Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17353 2022-11-23T02:42:49.3625682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17354 2022-11-23T02:42:49.3626091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3626266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3626689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3626880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3627116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3627495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3627671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3628084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3628352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3628585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3628852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeylrnvud 2022-11-23T02:42:49.3629133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeylrnvud/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3629394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkhvzsh1 2022-11-23T02:42:49.3629669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkhvzsh1/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3629911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3630144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3630383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3630615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3630708Z ok (6.218s) 2022-11-23T02:42:49.3630716Z 2022-11-23T02:42:49.3631051Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3631161Z Ran 1 test in 6.219s 2022-11-23T02:42:49.3631167Z 2022-11-23T02:42:49.3631256Z OK 2022-11-23T02:42:49.3631262Z 2022-11-23T02:42:49.3631382Z Generating XML reports... 2022-11-23T02:42:49.3631900Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023309.xml 2022-11-23T02:42:49.3632242Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3632647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3632815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3633230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3633427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3633434Z 2022-11-23T02:42:49.3633539Z Running tests... 2022-11-23T02:42:49.3633825Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3634093Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3634403Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17551 2022-11-23T02:42:49.3634630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17552 2022-11-23T02:42:49.3635028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3635209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3635615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3635806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3636046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3636445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3636622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3637025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3637219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3637510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3637777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphceel6fc 2022-11-23T02:42:49.3638058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphceel6fc/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3638320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9eeizea3 2022-11-23T02:42:49.3638585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9eeizea3/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3638829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3639065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3639305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3639543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3639784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3640017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3640360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3640594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3640691Z ok (6.218s) 2022-11-23T02:42:49.3640698Z 2022-11-23T02:42:49.3640995Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3641108Z Ran 1 test in 6.219s 2022-11-23T02:42:49.3641115Z 2022-11-23T02:42:49.3641202Z OK 2022-11-23T02:42:49.3641208Z 2022-11-23T02:42:49.3641329Z Generating XML reports... 2022-11-23T02:42:49.3641841Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023318.xml 2022-11-23T02:42:49.3642183Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3642592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3642773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3643193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3643389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3643395Z 2022-11-23T02:42:49.3643501Z Running tests... 2022-11-23T02:42:49.3643775Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3644148Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17749 2022-11-23T02:42:49.3644377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17750 2022-11-23T02:42:49.3644789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3644968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3645387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3645581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3645819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3646209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3646386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3646791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3647049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3647283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3647556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmbjh1_i 2022-11-23T02:42:49.3647845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmbjh1_i/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3648106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmm477sqz 2022-11-23T02:42:49.3648383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmm477sqz/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3648481Z ok (4.414s) 2022-11-23T02:42:49.3648488Z 2022-11-23T02:42:49.3648779Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3648891Z Ran 1 test in 4.415s 2022-11-23T02:42:49.3648897Z 2022-11-23T02:42:49.3648989Z OK 2022-11-23T02:42:49.3648995Z 2022-11-23T02:42:49.3649115Z Generating XML reports... 2022-11-23T02:42:49.3649608Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023328.xml 2022-11-23T02:42:49.3649941Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3650397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3650579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3650994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3651194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3651202Z 2022-11-23T02:42:49.3651305Z Running tests... 2022-11-23T02:42:49.3651593Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3651980Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17945 2022-11-23T02:42:49.3652207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17946 2022-11-23T02:42:49.3652610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3652788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3653194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3653389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3653623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3654024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3654206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3654611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3654807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3655036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3655305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy54l8b5u 2022-11-23T02:42:49.3655582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy54l8b5u/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3655826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoe60mbtq 2022-11-23T02:42:49.3656103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoe60mbtq/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3656249Z ok (4.089s) 2022-11-23T02:42:49.3656255Z 2022-11-23T02:42:49.3656548Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3656654Z Ran 1 test in 4.090s 2022-11-23T02:42:49.3656661Z 2022-11-23T02:42:49.3656746Z OK 2022-11-23T02:42:49.3656752Z 2022-11-23T02:42:49.3656872Z Generating XML reports... 2022-11-23T02:42:49.3657384Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023335.xml 2022-11-23T02:42:49.3657720Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3658123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3658300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3658709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3658912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3658918Z 2022-11-23T02:42:49.3659024Z Running tests... 2022-11-23T02:42:49.3659303Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3659736Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18141 2022-11-23T02:42:49.3659962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18142 2022-11-23T02:42:49.3660367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3660541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3660950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3661143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3661380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3661760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3661938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3662342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3662531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3662765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3663031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xf2ijhc 2022-11-23T02:42:49.3663306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xf2ijhc/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3663567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuzf0gzs 2022-11-23T02:42:49.3663844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuzf0gzs/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3663941Z ok (4.188s) 2022-11-23T02:42:49.3663948Z 2022-11-23T02:42:49.3664238Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3664343Z Ran 1 test in 4.188s 2022-11-23T02:42:49.3664350Z 2022-11-23T02:42:49.3664433Z OK 2022-11-23T02:42:49.3664440Z 2022-11-23T02:42:49.3664558Z Generating XML reports... 2022-11-23T02:42:49.3665060Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023342.xml 2022-11-23T02:42:49.3665397Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3665802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3666020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3666437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3666627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3666637Z 2022-11-23T02:42:49.3666739Z Running tests... 2022-11-23T02:42:49.3667009Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3667251Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3667613Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18337 2022-11-23T02:42:49.3667839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18338 2022-11-23T02:42:49.3668240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3668419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3668827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3669067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3669301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3669695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3669871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3670281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3670470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3670706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3670972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzcg1cme 2022-11-23T02:42:49.3671253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzcg1cme/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3671513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphpjy3i09 2022-11-23T02:42:49.3671788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphpjy3i09/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3671885Z ok (4.035s) 2022-11-23T02:42:49.3671891Z 2022-11-23T02:42:49.3672181Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3672287Z Ran 1 test in 4.036s 2022-11-23T02:42:49.3672294Z 2022-11-23T02:42:49.3672377Z OK 2022-11-23T02:42:49.3672384Z 2022-11-23T02:42:49.3672487Z Generating XML reports... 2022-11-23T02:42:49.3672996Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023349.xml 2022-11-23T02:42:49.3673338Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3673746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3674109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3674567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3674820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3674827Z 2022-11-23T02:42:49.3674972Z Running tests... 2022-11-23T02:42:49.3675306Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3675590Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3676049Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18533 2022-11-23T02:42:49.3676358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18534 2022-11-23T02:42:49.3676910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3677134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3677601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3677838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3678101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3692957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3693228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3693739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3694103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3694349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3694626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnv73u5gx 2022-11-23T02:42:49.3694907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnv73u5gx/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3695175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnouizha 2022-11-23T02:42:49.3695462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnouizha/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3695566Z ok (4.112s) 2022-11-23T02:42:49.3695578Z 2022-11-23T02:42:49.3695888Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3695998Z Ran 1 test in 4.113s 2022-11-23T02:42:49.3696007Z 2022-11-23T02:42:49.3696091Z OK 2022-11-23T02:42:49.3696097Z 2022-11-23T02:42:49.3696218Z Generating XML reports... 2022-11-23T02:42:49.3696740Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023356.xml 2022-11-23T02:42:49.3697086Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3697480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3697663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3698079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3698281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3698288Z 2022-11-23T02:42:49.3698393Z Running tests... 2022-11-23T02:42:49.3698684Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3699053Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18729 2022-11-23T02:42:49.3699284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18730 2022-11-23T02:42:49.3699681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3699858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3700270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3700462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3700771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3701172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3701356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3701765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3701962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3702196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3702350Z skip: Need at least 4 CUDA devices (3.424s) 2022-11-23T02:42:49.3702357Z 2022-11-23T02:42:49.3702643Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3702750Z Ran 1 test in 3.425s 2022-11-23T02:42:49.3702760Z 2022-11-23T02:42:49.3702864Z OK (skipped=1) 2022-11-23T02:42:49.3702870Z 2022-11-23T02:42:49.3702975Z Generating XML reports... 2022-11-23T02:42:49.3703493Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023403.xml 2022-11-23T02:42:49.3703879Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3704280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3704459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3704875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3705075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3705082Z 2022-11-23T02:42:49.3705187Z Running tests... 2022-11-23T02:42:49.3705482Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3705692Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3706028Z Tests that DDP with ``device_ids`` specified can run a forward and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18906 2022-11-23T02:42:49.3706259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18907 2022-11-23T02:42:49.3706663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3706840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3707256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3707451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3707690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3707954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3708357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3708538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3708954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3709145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3709383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3709620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3710059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3710573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3710841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2qaws5q 2022-11-23T02:42:49.3711129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2qaws5q/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3711397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnujbe5yw 2022-11-23T02:42:49.3711679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnujbe5yw/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3711782Z ok (7.623s) 2022-11-23T02:42:49.3711790Z 2022-11-23T02:42:49.3712087Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3712197Z Ran 1 test in 7.624s 2022-11-23T02:42:49.3712204Z 2022-11-23T02:42:49.3712291Z OK 2022-11-23T02:42:49.3712298Z 2022-11-23T02:42:49.3712425Z Generating XML reports... 2022-11-23T02:42:49.3712939Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023410.xml 2022-11-23T02:42:49.3713281Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3713740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3713923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3714341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3714538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3714546Z 2022-11-23T02:42:49.3714653Z Running tests... 2022-11-23T02:42:49.3714949Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3715297Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19104 2022-11-23T02:42:49.3715693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19105 2022-11-23T02:42:49.3716084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3716267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3716684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3716879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3717119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3717518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3717693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3718106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3718300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3718538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3718812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0cgzelo 2022-11-23T02:42:49.3719098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0cgzelo/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3719360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1h_j699 2022-11-23T02:42:49.3719635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1h_j699/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3719877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3720264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3720503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3720733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3720977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3721221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3721455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3721680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3721760Z ok (4.417s) 2022-11-23T02:42:49.3721768Z 2022-11-23T02:42:49.3722063Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3722173Z Ran 1 test in 4.417s 2022-11-23T02:42:49.3722180Z 2022-11-23T02:42:49.3722266Z OK 2022-11-23T02:42:49.3722272Z 2022-11-23T02:42:49.3722394Z Generating XML reports... 2022-11-23T02:42:49.3722915Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023420.xml 2022-11-23T02:42:49.3723316Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3723727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3723905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3724316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3724509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3724516Z 2022-11-23T02:42:49.3724620Z Running tests... 2022-11-23T02:42:49.3724917Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3725268Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19300 2022-11-23T02:42:49.3725499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19301 2022-11-23T02:42:49.3725895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3726070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3726475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3726676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3726913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3727514Z /opt/conda/lib/python3.8/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-11-23T02:42:49.3727814Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-11-23T02:42:49.3728081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7touj9x 2022-11-23T02:42:49.3728346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7touj9x/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3728749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3728929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3729348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3729541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3729833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3730429Z /opt/conda/lib/python3.8/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-11-23T02:42:49.3730736Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-11-23T02:42:49.3731003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fq_h_55 2022-11-23T02:42:49.3731278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fq_h_55/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3731376Z ok (3.512s) 2022-11-23T02:42:49.3731383Z 2022-11-23T02:42:49.3731673Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3731779Z Ran 1 test in 3.512s 2022-11-23T02:42:49.3731786Z 2022-11-23T02:42:49.3731870Z OK 2022-11-23T02:42:49.3731879Z 2022-11-23T02:42:49.3731997Z Generating XML reports... 2022-11-23T02:42:49.3732511Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023427.xml 2022-11-23T02:42:49.3732850Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3733307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3733492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3733908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3734101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3734108Z 2022-11-23T02:42:49.3734212Z Running tests... 2022-11-23T02:42:49.3734483Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3734837Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19481 2022-11-23T02:42:49.3735068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19482 2022-11-23T02:42:49.3735497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3735674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3736091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3736286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3736525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3736928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3737108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3737554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3737747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3737987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3738256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83xl_cwk 2022-11-23T02:42:49.3738532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83xl_cwk/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3738792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9abgokbq 2022-11-23T02:42:49.3739087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9abgokbq/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3739184Z ok (4.219s) 2022-11-23T02:42:49.3739239Z 2022-11-23T02:42:49.3739530Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3739635Z Ran 1 test in 4.220s 2022-11-23T02:42:49.3739641Z 2022-11-23T02:42:49.3739721Z OK 2022-11-23T02:42:49.3739728Z 2022-11-23T02:42:49.3739864Z Generating XML reports... 2022-11-23T02:42:49.3740375Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023434.xml 2022-11-23T02:42:49.3740714Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3741135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3741311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3741722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3741922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3741932Z 2022-11-23T02:42:49.3742035Z Running tests... 2022-11-23T02:42:49.3742338Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3742752Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19677 2022-11-23T02:42:49.3742986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19678 2022-11-23T02:42:49.3743408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3743583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3743994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3744187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3744440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3744842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3745015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3745461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3745651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3745883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3746159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpezxozc 2022-11-23T02:42:49.3746457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpezxozc/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3746703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6spcvo6x 2022-11-23T02:42:49.3746982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6spcvo6x/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3747078Z ok (4.013s) 2022-11-23T02:42:49.3747085Z 2022-11-23T02:42:49.3747385Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3747492Z Ran 1 test in 4.013s 2022-11-23T02:42:49.3747499Z 2022-11-23T02:42:49.3747605Z OK 2022-11-23T02:42:49.3747611Z 2022-11-23T02:42:49.3747730Z Generating XML reports... 2022-11-23T02:42:49.3748237Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023441.xml 2022-11-23T02:42:49.3748580Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3749014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3749243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3749661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3749873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3749879Z 2022-11-23T02:42:49.3749988Z Running tests... 2022-11-23T02:42:49.3750287Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3750637Z test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19873 2022-11-23T02:42:49.3750861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19874 2022-11-23T02:42:49.3751276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3751451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3751882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3752075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3752307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3752738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3752917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3753329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3753527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3753780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3754048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgpuk8tm 2022-11-23T02:42:49.3754335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgpuk8tm/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3754598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tpts68d 2022-11-23T02:42:49.3754979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tpts68d/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3755163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3755354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3755534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3755730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3755809Z ok (6.841s) 2022-11-23T02:42:49.3755814Z 2022-11-23T02:42:49.3756037Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3756126Z Ran 1 test in 6.841s 2022-11-23T02:42:49.3756132Z 2022-11-23T02:42:49.3756196Z OK 2022-11-23T02:42:49.3756200Z 2022-11-23T02:42:49.3756291Z Generating XML reports... 2022-11-23T02:42:49.3756687Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023448.xml 2022-11-23T02:42:49.3756945Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3757245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3757383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3757694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3757837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3757882Z 2022-11-23T02:42:49.3757963Z Running tests... 2022-11-23T02:42:49.3758187Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3759031Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82632 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.483s) 2022-11-23T02:42:49.3759054Z 2022-11-23T02:42:49.3759257Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3759341Z Ran 1 test in 0.483s 2022-11-23T02:42:49.3759347Z 2022-11-23T02:42:49.3759430Z OK (skipped=1) 2022-11-23T02:42:49.3759434Z 2022-11-23T02:42:49.3759526Z Generating XML reports... 2022-11-23T02:42:49.3759911Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023458.xml 2022-11-23T02:42:49.3760238Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3760734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3760926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3761254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3761403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3761408Z 2022-11-23T02:42:49.3761491Z Running tests... 2022-11-23T02:42:49.3761739Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3762552Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/83301 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.482s) 2022-11-23T02:42:49.3762562Z 2022-11-23T02:42:49.3762786Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3762869Z Ran 1 test in 0.482s 2022-11-23T02:42:49.3762875Z 2022-11-23T02:42:49.3762956Z OK (skipped=1) 2022-11-23T02:42:49.3762960Z 2022-11-23T02:42:49.3763056Z Generating XML reports... 2022-11-23T02:42:49.3763448Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023501.xml 2022-11-23T02:42:49.3763709Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3764021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3764162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3764483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3764633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3764637Z 2022-11-23T02:42:49.3764735Z Running tests... 2022-11-23T02:42:49.3764943Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3765788Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82385 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.493s) 2022-11-23T02:42:49.3765809Z 2022-11-23T02:42:49.3766043Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3766168Z Ran 1 test in 0.493s 2022-11-23T02:42:49.3766185Z 2022-11-23T02:42:49.3766250Z OK (skipped=1) 2022-11-23T02:42:49.3766254Z 2022-11-23T02:42:49.3766348Z Generating XML reports... 2022-11-23T02:42:49.3766757Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023504.xml 2022-11-23T02:42:49.3767023Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3767337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3767492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3767810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3767966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3767973Z 2022-11-23T02:42:49.3768059Z Running tests... 2022-11-23T02:42:49.3768281Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3769167Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82979 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.496s) 2022-11-23T02:42:49.3769174Z 2022-11-23T02:42:49.3769400Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3769482Z Ran 1 test in 0.496s 2022-11-23T02:42:49.3769487Z 2022-11-23T02:42:49.3769570Z OK (skipped=1) 2022-11-23T02:42:49.3769575Z 2022-11-23T02:42:49.3769672Z Generating XML reports... 2022-11-23T02:42:49.3770078Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023508.xml 2022-11-23T02:42:49.3770347Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3770659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3770819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3771139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3771291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3771296Z 2022-11-23T02:42:49.3771375Z Running tests... 2022-11-23T02:42:49.3771602Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3772532Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82400 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.478s) 2022-11-23T02:42:49.3772540Z 2022-11-23T02:42:49.3772756Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3772853Z Ran 1 test in 0.478s 2022-11-23T02:42:49.3772858Z 2022-11-23T02:42:49.3772934Z OK (skipped=1) 2022-11-23T02:42:49.3772938Z 2022-11-23T02:42:49.3773028Z Generating XML reports... 2022-11-23T02:42:49.3773420Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023511.xml 2022-11-23T02:42:49.3773659Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3773960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3774152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3774460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3774607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3774615Z 2022-11-23T02:42:49.3774698Z Running tests... 2022-11-23T02:42:49.3774914Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3775732Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82500 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.483s) 2022-11-23T02:42:49.3775738Z 2022-11-23T02:42:49.3775958Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3776043Z Ran 1 test in 0.483s 2022-11-23T02:42:49.3776047Z 2022-11-23T02:42:49.3776137Z OK (skipped=1) 2022-11-23T02:42:49.3776142Z 2022-11-23T02:42:49.3776233Z Generating XML reports... 2022-11-23T02:42:49.3776674Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023515.xml 2022-11-23T02:42:49.3776926Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3777231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3777363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3777682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3777827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3777836Z 2022-11-23T02:42:49.3777915Z Running tests... 2022-11-23T02:42:49.3778143Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3778371Z test_fp16 (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20430 2022-11-23T02:42:49.3778552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20431 2022-11-23T02:42:49.3778857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3778992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3779317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3779462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3779646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3779946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3780106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3780414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3780561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3780738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3780937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7s58xleu 2022-11-23T02:42:49.3781148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7s58xleu/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3781348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt60tdxop 2022-11-23T02:42:49.3781604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt60tdxop/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3781679Z ok (6.419s) 2022-11-23T02:42:49.3781684Z 2022-11-23T02:42:49.3781903Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3781985Z Ran 1 test in 6.420s 2022-11-23T02:42:49.3781992Z 2022-11-23T02:42:49.3782060Z OK 2022-11-23T02:42:49.3782064Z 2022-11-23T02:42:49.3782153Z Generating XML reports... 2022-11-23T02:42:49.3782516Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023518.xml 2022-11-23T02:42:49.3782782Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3783085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3783220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3783536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3783685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3783691Z 2022-11-23T02:42:49.3783769Z Running tests... 2022-11-23T02:42:49.3784034Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3784299Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20628 2022-11-23T02:42:49.3784467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20629 2022-11-23T02:42:49.3784773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3784907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3785225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3785371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3785548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3785999Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3786303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3786437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3786756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3786900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3787074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3787529Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3787730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlslg6cp 2022-11-23T02:42:49.3787936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlslg6cp/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3788136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5k2a3o26 2022-11-23T02:42:49.3788382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5k2a3o26/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3788457Z ok (4.526s) 2022-11-23T02:42:49.3788462Z 2022-11-23T02:42:49.3788696Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3788765Z Ran 1 test in 4.527s 2022-11-23T02:42:49.3788786Z 2022-11-23T02:42:49.3788841Z OK 2022-11-23T02:42:49.3788846Z 2022-11-23T02:42:49.3788939Z Generating XML reports... 2022-11-23T02:42:49.3789320Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023527.xml 2022-11-23T02:42:49.3789591Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3789890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3790024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3790333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3790480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3790486Z 2022-11-23T02:42:49.3790563Z Running tests... 2022-11-23T02:42:49.3790838Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3791105Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20824 2022-11-23T02:42:49.3791288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20825 2022-11-23T02:42:49.3791597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3791730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3792048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3792195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3792377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3792822Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3793139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3793271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3793581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3793728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3793907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3794352Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3794557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyq8i9623 2022-11-23T02:42:49.3794769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyq8i9623/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3794951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdno0d6v5 2022-11-23T02:42:49.3795158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdno0d6v5/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3795277Z ok (4.229s) 2022-11-23T02:42:49.3795283Z 2022-11-23T02:42:49.3795502Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3795585Z Ran 1 test in 4.229s 2022-11-23T02:42:49.3795590Z 2022-11-23T02:42:49.3795654Z OK 2022-11-23T02:42:49.3795661Z 2022-11-23T02:42:49.3795753Z Generating XML reports... 2022-11-23T02:42:49.3796135Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023535.xml 2022-11-23T02:42:49.3796405Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3796710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3796844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3797156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3797303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3797308Z 2022-11-23T02:42:49.3797384Z Running tests... 2022-11-23T02:42:49.3797613Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3797903Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21020 2022-11-23T02:42:49.3798079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21021 2022-11-23T02:42:49.3798380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3798513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3798822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3798976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3799155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3799437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3799570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3799876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3800019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3800251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3800628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9r5tl4cd 2022-11-23T02:42:49.3800863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9r5tl4cd/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3801067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0elts1r 2022-11-23T02:42:49.3801283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0elts1r/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3801359Z ok (6.127s) 2022-11-23T02:42:49.3801366Z 2022-11-23T02:42:49.3801609Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3801695Z Ran 1 test in 6.128s 2022-11-23T02:42:49.3801700Z 2022-11-23T02:42:49.3801768Z OK 2022-11-23T02:42:49.3801772Z 2022-11-23T02:42:49.3801866Z Generating XML reports... 2022-11-23T02:42:49.3802263Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023542.xml 2022-11-23T02:42:49.3802519Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3802843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3803033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3803357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3803512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3803518Z 2022-11-23T02:42:49.3803599Z Running tests... 2022-11-23T02:42:49.3803819Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3804122Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21218 2022-11-23T02:42:49.3804301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21219 2022-11-23T02:42:49.3804609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3804752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3805082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3805233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3805463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3805773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3805911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3806239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3806390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3806571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3806790Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:42:49.3807008Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:42:49.3807214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyredzwzq 2022-11-23T02:42:49.3807438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyredzwzq/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3807644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp123x_t3m 2022-11-23T02:42:49.3807889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp123x_t3m/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3808077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3808262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3808448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3808630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3808825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3809026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3809192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3809373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3809548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3809730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3809949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3810128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3810303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3810482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3810664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3810843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3811035Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3811212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3811499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3811677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3811866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3812078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3812252Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3812427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3812603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3812777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3812967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3813129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3813307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3813484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3813659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3813843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3814012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3814182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3814357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3814531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3814609Z ok (8.231s) 2022-11-23T02:42:49.3814614Z 2022-11-23T02:42:49.3814851Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3814935Z Ran 1 test in 8.232s 2022-11-23T02:42:49.3814940Z 2022-11-23T02:42:49.3815006Z OK 2022-11-23T02:42:49.3815010Z 2022-11-23T02:42:49.3815100Z Generating XML reports... 2022-11-23T02:42:49.3815491Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023551.xml 2022-11-23T02:42:49.3815748Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3816046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3816182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3816490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3816690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3816694Z 2022-11-23T02:42:49.3816777Z Running tests... 2022-11-23T02:42:49.3816985Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3817259Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21416 2022-11-23T02:42:49.3817434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21417 2022-11-23T02:42:49.3817736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3817870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3818185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3818358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3818536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3818835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3818968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3819313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3819458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3819638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3819755Z skip: Need at least 4 CUDA devices (3.614s) 2022-11-23T02:42:49.3819760Z 2022-11-23T02:42:49.3819977Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3820054Z Ran 1 test in 3.614s 2022-11-23T02:42:49.3820062Z 2022-11-23T02:42:49.3820141Z OK (skipped=1) 2022-11-23T02:42:49.3820145Z 2022-11-23T02:42:49.3820239Z Generating XML reports... 2022-11-23T02:42:49.3820631Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023602.xml 2022-11-23T02:42:49.3820894Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3821193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3821320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3821613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3821756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3821761Z 2022-11-23T02:42:49.3821840Z Running tests... 2022-11-23T02:42:49.3822057Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3822323Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21593 2022-11-23T02:42:49.3822508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21594 2022-11-23T02:42:49.3822817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3822955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3823266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3823413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3823593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3824037Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3824616Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3825055Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3825481Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3825950Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3826402Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3826715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3826852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3827167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3827316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3827498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3827939Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3828378Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3828816Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3829265Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3829774Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3830201Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3830275Z ok (3.691s) 2022-11-23T02:42:49.3830281Z 2022-11-23T02:42:49.3830518Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3830601Z Ran 1 test in 3.692s 2022-11-23T02:42:49.3830606Z 2022-11-23T02:42:49.3830672Z OK 2022-11-23T02:42:49.3830677Z 2022-11-23T02:42:49.3830771Z Generating XML reports... 2022-11-23T02:42:49.3831139Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023609.xml 2022-11-23T02:42:49.3831436Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3831738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3831872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3832187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3832334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3832339Z 2022-11-23T02:42:49.3832419Z Running tests... 2022-11-23T02:42:49.3832641Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3832921Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21770 2022-11-23T02:42:49.3833096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21771 2022-11-23T02:42:49.3833398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3833534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3833844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3833990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3834169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3834481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3834614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3834925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3835087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3835266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3835470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxfae9uj 2022-11-23T02:42:49.3835684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxfae9uj/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3835883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxq3ndtaf 2022-11-23T02:42:49.3836079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxq3ndtaf/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3836209Z ok (6.424s) 2022-11-23T02:42:49.3836215Z 2022-11-23T02:42:49.3836433Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3836512Z Ran 1 test in 6.432s 2022-11-23T02:42:49.3836517Z 2022-11-23T02:42:49.3836584Z OK 2022-11-23T02:42:49.3836590Z 2022-11-23T02:42:49.3836679Z Generating XML reports... 2022-11-23T02:42:49.3837065Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023616.xml 2022-11-23T02:42:49.3837333Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3837639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3837776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3838096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3838244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3838249Z 2022-11-23T02:42:49.3838330Z Running tests... 2022-11-23T02:42:49.3838553Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3838879Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21968 2022-11-23T02:42:49.3839051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21969 2022-11-23T02:42:49.3839352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3839491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3839804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3839952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3840201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3840679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3840802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3841120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3841283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3841461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3841663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6m_39yf 2022-11-23T02:42:49.3841884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6m_39yf/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3842089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpsowaj1 2022-11-23T02:42:49.3842305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpsowaj1/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3842383Z ok (6.119s) 2022-11-23T02:42:49.3842388Z 2022-11-23T02:42:49.3842611Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3842694Z Ran 1 test in 6.120s 2022-11-23T02:42:49.3842699Z 2022-11-23T02:42:49.3842766Z OK 2022-11-23T02:42:49.3842771Z 2022-11-23T02:42:49.3842865Z Generating XML reports... 2022-11-23T02:42:49.3843268Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023625.xml 2022-11-23T02:42:49.3843525Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3843886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3844026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3844345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3844687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3844692Z 2022-11-23T02:42:49.3844774Z Running tests... 2022-11-23T02:42:49.3844995Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3845295Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22166 2022-11-23T02:42:49.3845470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22167 2022-11-23T02:42:49.3845778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3845921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3846241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3846432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3846613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3846925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3847060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3847374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3847525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3847713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3847917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_n8zy_3 2022-11-23T02:42:49.3848130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_n8zy_3/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3848332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24gaqrbz 2022-11-23T02:42:49.3848542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24gaqrbz/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3848730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3848917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3849008Z ok (6.641s) 2022-11-23T02:42:49.3849013Z 2022-11-23T02:42:49.3849237Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3849323Z Ran 1 test in 6.642s 2022-11-23T02:42:49.3849328Z 2022-11-23T02:42:49.3849397Z OK 2022-11-23T02:42:49.3849402Z 2022-11-23T02:42:49.3849482Z Generating XML reports... 2022-11-23T02:42:49.3849876Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023634.xml 2022-11-23T02:42:49.3850142Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3850451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3850591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3850909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3851072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3851077Z 2022-11-23T02:42:49.3851154Z Running tests... 2022-11-23T02:42:49.3851417Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3851719Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22364 2022-11-23T02:42:49.3851892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22365 2022-11-23T02:42:49.3852310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3852456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3852765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3852909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3853080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3853379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3853523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3853871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3854016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3854204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3854406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfk0q1ua 2022-11-23T02:42:49.3854619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfk0q1ua/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3854804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprjabnr5 2022-11-23T02:42:49.3855021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprjabnr5/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3855218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3855411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3855488Z ok (6.220s) 2022-11-23T02:42:49.3855493Z 2022-11-23T02:42:49.3855715Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3855798Z Ran 1 test in 6.220s 2022-11-23T02:42:49.3855803Z 2022-11-23T02:42:49.3855872Z OK 2022-11-23T02:42:49.3855877Z 2022-11-23T02:42:49.3855965Z Generating XML reports... 2022-11-23T02:42:49.3856342Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023644.xml 2022-11-23T02:42:49.3856594Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3856897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3857033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3857347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3857500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3857505Z 2022-11-23T02:42:49.3857584Z Running tests... 2022-11-23T02:42:49.3857798Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3858061Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22562 2022-11-23T02:42:49.3858243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22563 2022-11-23T02:42:49.3858539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3858716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3859032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3859166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3859349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3859646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3859783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3860092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3860238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3860417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3860531Z skip: Need at least 4 CUDA devices (3.534s) 2022-11-23T02:42:49.3860536Z 2022-11-23T02:42:49.3860752Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3860834Z Ran 1 test in 3.535s 2022-11-23T02:42:49.3860840Z 2022-11-23T02:42:49.3860956Z OK (skipped=1) 2022-11-23T02:42:49.3860961Z 2022-11-23T02:42:49.3861051Z Generating XML reports... 2022-11-23T02:42:49.3861426Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023653.xml 2022-11-23T02:42:49.3861683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3861982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3862116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3862426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3862581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3862586Z 2022-11-23T02:42:49.3862664Z Running tests... 2022-11-23T02:42:49.3862883Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3863144Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22739 2022-11-23T02:42:49.3863330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22740 2022-11-23T02:42:49.3863629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3863750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3864061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3864208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3864388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3864684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3864817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3865123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3865270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3865443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3865558Z skip: Need at least 8 CUDA devices (3.644s) 2022-11-23T02:42:49.3865563Z 2022-11-23T02:42:49.3865794Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3865920Z Ran 1 test in 3.645s 2022-11-23T02:42:49.3865925Z 2022-11-23T02:42:49.3866002Z OK (skipped=1) 2022-11-23T02:42:49.3866006Z 2022-11-23T02:42:49.3866104Z Generating XML reports... 2022-11-23T02:42:49.3866488Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023659.xml 2022-11-23T02:42:49.3866740Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3867040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3867169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3867486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3867647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3867654Z 2022-11-23T02:42:49.3867737Z Running tests... 2022-11-23T02:42:49.3867941Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3868263Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22916 2022-11-23T02:42:49.3868433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22917 2022-11-23T02:42:49.3868728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3868862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3869172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3869315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3869493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3869790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3869919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3870225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3870376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3870549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3870622Z ok (3.709s) 2022-11-23T02:42:49.3870627Z 2022-11-23T02:42:49.3870842Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3870926Z Ran 1 test in 3.710s 2022-11-23T02:42:49.3870930Z 2022-11-23T02:42:49.3871002Z OK 2022-11-23T02:42:49.3871006Z 2022-11-23T02:42:49.3871100Z Generating XML reports... 2022-11-23T02:42:49.3871483Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023706.xml 2022-11-23T02:42:49.3871737Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3872045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3872196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3872683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3872820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3872840Z 2022-11-23T02:42:49.3872908Z Running tests... 2022-11-23T02:42:49.3873129Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3873436Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23097 2022-11-23T02:42:49.3873655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23098 2022-11-23T02:42:49.3873965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3874105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3874419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3874569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3874750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3875055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3875194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3875508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3875658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3875872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3875990Z skip: Need at least 4 CUDA devices (3.516s) 2022-11-23T02:42:49.3875995Z 2022-11-23T02:42:49.3876222Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3876309Z Ran 1 test in 3.516s 2022-11-23T02:42:49.3876314Z 2022-11-23T02:42:49.3876395Z OK (skipped=1) 2022-11-23T02:42:49.3876400Z 2022-11-23T02:42:49.3876498Z Generating XML reports... 2022-11-23T02:42:49.3876892Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023712.xml 2022-11-23T02:42:49.3877153Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3877446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3877583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3877898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3878049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3878054Z 2022-11-23T02:42:49.3878137Z Running tests... 2022-11-23T02:42:49.3878359Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3878667Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23274 2022-11-23T02:42:49.3878844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23275 2022-11-23T02:42:49.3879163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3879306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3879622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3879776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3879957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3880368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3880505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3880813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3881021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3881205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3881418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrli1qkx 2022-11-23T02:42:49.3881655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrli1qkx/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3881858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl23w8u0f 2022-11-23T02:42:49.3882077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl23w8u0f/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3882280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3882455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3882538Z ok (6.218s) 2022-11-23T02:42:49.3882544Z 2022-11-23T02:42:49.3882828Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3882914Z Ran 1 test in 6.219s 2022-11-23T02:42:49.3882919Z 2022-11-23T02:42:49.3882986Z OK 2022-11-23T02:42:49.3882990Z 2022-11-23T02:42:49.3883089Z Generating XML reports... 2022-11-23T02:42:49.3883542Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023719.xml 2022-11-23T02:42:49.3883824Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3884139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3884279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3884596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3884751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3884757Z 2022-11-23T02:42:49.3884837Z Running tests... 2022-11-23T02:42:49.3885077Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3885377Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23472 2022-11-23T02:42:49.3885550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23473 2022-11-23T02:42:49.3885859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3885995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3886313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3886461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3886647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3886950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3887076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3887402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3887554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3887738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3887947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8fskksw 2022-11-23T02:42:49.3888164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8fskksw/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3888424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbee0ncs 2022-11-23T02:42:49.3888637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbee0ncs/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3888825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3889016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3889092Z ok (6.217s) 2022-11-23T02:42:49.3889097Z 2022-11-23T02:42:49.3889326Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3889413Z Ran 1 test in 6.218s 2022-11-23T02:42:49.3889418Z 2022-11-23T02:42:49.3889486Z OK 2022-11-23T02:42:49.3889491Z 2022-11-23T02:42:49.3889586Z Generating XML reports... 2022-11-23T02:42:49.3889979Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023728.xml 2022-11-23T02:42:49.3890241Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3890554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3890691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3891153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3891301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3891306Z 2022-11-23T02:42:49.3891387Z Running tests... 2022-11-23T02:42:49.3891602Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3891872Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23670 2022-11-23T02:42:49.3892042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23671 2022-11-23T02:42:49.3892355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3892487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3892793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3892943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3893119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3893425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3893559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3893865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3894016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3894190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3894265Z ok (19.054s) 2022-11-23T02:42:49.3894271Z 2022-11-23T02:42:49.3894491Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3894575Z Ran 1 test in 19.055s 2022-11-23T02:42:49.3894580Z 2022-11-23T02:42:49.3894645Z OK 2022-11-23T02:42:49.3894650Z 2022-11-23T02:42:49.3894741Z Generating XML reports... 2022-11-23T02:42:49.3895115Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023737.xml 2022-11-23T02:42:49.3895366Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3895664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3895841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3896139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3896292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3896311Z 2022-11-23T02:42:49.3896380Z Running tests... 2022-11-23T02:42:49.3896591Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3896734Z test_no_grad (__main__.DistributedDataParallelTest) 2022-11-23T02:42:49.3896982Z Note: this test can be sped up by only running it on a CPU module ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23862 2022-11-23T02:42:49.3897170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23863 2022-11-23T02:42:49.3897469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3897608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3897914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3898058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3898279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3898576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3898713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3899014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3899155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3899330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3899546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2m4rvm4 2022-11-23T02:42:49.3899753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2m4rvm4/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3899951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv_3nqgg 2022-11-23T02:42:49.3900154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv_3nqgg/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3900228Z ok (6.153s) 2022-11-23T02:42:49.3900233Z 2022-11-23T02:42:49.3900439Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3900526Z Ran 1 test in 6.153s 2022-11-23T02:42:49.3900531Z 2022-11-23T02:42:49.3900600Z OK 2022-11-23T02:42:49.3900604Z 2022-11-23T02:42:49.3900697Z Generating XML reports... 2022-11-23T02:42:49.3901080Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023759.xml 2022-11-23T02:42:49.3901340Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3901641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3901777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3902083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3902229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3902233Z 2022-11-23T02:42:49.3902312Z Running tests... 2022-11-23T02:42:49.3902532Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3902797Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24052 2022-11-23T02:42:49.3903003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24053 2022-11-23T02:42:49.3903306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3903438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3903746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3903890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3904070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3904367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3904500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3904810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3904944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3905115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3905354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8ftjz57 2022-11-23T02:42:49.3905568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8ftjz57/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3905764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0f8481_ 2022-11-23T02:42:49.3905962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0f8481_/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3906039Z ok (3.927s) 2022-11-23T02:42:49.3906044Z 2022-11-23T02:42:49.3906273Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3906353Z Ran 1 test in 3.927s 2022-11-23T02:42:49.3906360Z 2022-11-23T02:42:49.3906423Z OK 2022-11-23T02:42:49.3906428Z 2022-11-23T02:42:49.3906517Z Generating XML reports... 2022-11-23T02:42:49.3906905Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023808.xml 2022-11-23T02:42:49.3907164Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3907465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3907597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3907910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3908054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3908059Z 2022-11-23T02:42:49.3908138Z Running tests... 2022-11-23T02:42:49.3908357Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3908611Z test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24240 2022-11-23T02:42:49.3908786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24241 2022-11-23T02:42:49.3909081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3909203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3909507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3909653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3909828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3910020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.3910371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3910508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3910818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3910965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3911143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3911334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.3911663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3911991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.3912070Z ok (3.511s) 2022-11-23T02:42:49.3912075Z 2022-11-23T02:42:49.3912307Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3912387Z Ran 1 test in 3.512s 2022-11-23T02:42:49.3912393Z 2022-11-23T02:42:49.3912457Z OK 2022-11-23T02:42:49.3912499Z 2022-11-23T02:42:49.3912590Z Generating XML reports... 2022-11-23T02:42:49.3912973Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023815.xml 2022-11-23T02:42:49.3913223Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3913526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3913662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3913974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3914110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3914132Z 2022-11-23T02:42:49.3914197Z Running tests... 2022-11-23T02:42:49.3914415Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3914685Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24421 2022-11-23T02:42:49.3914855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24422 2022-11-23T02:42:49.3915170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3915304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3915608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3915759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3915931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3916373Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3916687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3916829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3917138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3917323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3917501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3917934Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3918139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6341d5t8 2022-11-23T02:42:49.3918342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6341d5t8/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3918536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3w2gp4e 2022-11-23T02:42:49.3918749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3w2gp4e/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3919239Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3919675Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3920106Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3920784Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3921226Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3921655Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3922098Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3922533Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3922986Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3923466Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3923909Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3924389Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3924839Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3925272Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3925350Z ok (4.714s) 2022-11-23T02:42:49.3925356Z 2022-11-23T02:42:49.3925589Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3925672Z Ran 1 test in 4.715s 2022-11-23T02:42:49.3925678Z 2022-11-23T02:42:49.3925751Z OK 2022-11-23T02:42:49.3925756Z 2022-11-23T02:42:49.3925849Z Generating XML reports... 2022-11-23T02:42:49.3926238Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023822.xml 2022-11-23T02:42:49.3926502Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3926814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3926956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3927281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3927421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3927440Z 2022-11-23T02:42:49.3927508Z Running tests... 2022-11-23T02:42:49.3927736Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3928019Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24617 2022-11-23T02:42:49.3928193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24618 2022-11-23T02:42:49.3928506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3928643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3928972Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3929174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3929359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3929809Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3930120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3930257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3930589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3930738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3930919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3931418Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3931622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtf75dnq 2022-11-23T02:42:49.3931941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtf75dnq/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3932137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0mfrjh3 2022-11-23T02:42:49.3932341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0mfrjh3/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3932775Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3933211Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3933638Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3934098Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3934635Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3935063Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3935538Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3935996Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3936517Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3937025Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3949033Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3949728Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T02:42:49.3950172Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3950594Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:42:49.3950673Z ok (4.120s) 2022-11-23T02:42:49.3950680Z 2022-11-23T02:42:49.3950955Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3951038Z Ran 1 test in 4.121s 2022-11-23T02:42:49.3951044Z 2022-11-23T02:42:49.3951107Z OK 2022-11-23T02:42:49.3951111Z 2022-11-23T02:42:49.3951199Z Generating XML reports... 2022-11-23T02:42:49.3951590Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023829.xml 2022-11-23T02:42:49.3951843Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3952149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3952289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3952749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3952884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3952896Z 2022-11-23T02:42:49.3952979Z Running tests... 2022-11-23T02:42:49.3953200Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3953467Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24813 2022-11-23T02:42:49.3953634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24814 2022-11-23T02:42:49.3953930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3954064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3954372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3954515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3954687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3955037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3955166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3955472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3955612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3955781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3955984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8kcsup0 2022-11-23T02:42:49.3956197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8kcsup0/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3956397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p6zab2i 2022-11-23T02:42:49.3956623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p6zab2i/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3956806Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3956987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3957058Z ok (7.621s) 2022-11-23T02:42:49.3957064Z 2022-11-23T02:42:49.3957269Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3957352Z Ran 1 test in 7.629s 2022-11-23T02:42:49.3957357Z 2022-11-23T02:42:49.3957422Z OK 2022-11-23T02:42:49.3957427Z 2022-11-23T02:42:49.3957515Z Generating XML reports... 2022-11-23T02:42:49.3957899Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023837.xml 2022-11-23T02:42:49.3958154Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3958461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3958595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3958898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3959040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3959046Z 2022-11-23T02:42:49.3959119Z Running tests... 2022-11-23T02:42:49.3959339Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3959611Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25050 2022-11-23T02:42:49.3959816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25051 2022-11-23T02:42:49.3960111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3960309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3960788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3960934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3961114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3961425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3961554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3961866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3962010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3962225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3962435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0l3mrjgw 2022-11-23T02:42:49.3962650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0l3mrjgw/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3962850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11ls2oyi 2022-11-23T02:42:49.3963060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11ls2oyi/_remote_module_non_scriptable.py 2022-11-23T02:42:49.3963245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3963433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:42:49.3963502Z ok (4.823s) 2022-11-23T02:42:49.3963507Z 2022-11-23T02:42:49.3963733Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3963813Z Ran 1 test in 4.823s 2022-11-23T02:42:49.3963821Z 2022-11-23T02:42:49.3963885Z OK 2022-11-23T02:42:49.3963890Z 2022-11-23T02:42:49.3963978Z Generating XML reports... 2022-11-23T02:42:49.3964363Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123023847.xml 2022-11-23T02:42:49.3964618Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3964929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3965064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3965381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3965529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3965534Z 2022-11-23T02:42:49.3965615Z Running tests... 2022-11-23T02:42:49.3965839Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3966097Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25246 2022-11-23T02:42:49.3966257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25247 2022-11-23T02:42:49.3966425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25248 2022-11-23T02:42:49.3966731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3966868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3967257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3967407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3967594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:42:49.3967895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3968030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3968341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3968490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3968667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3968972Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3969106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3969422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3969611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3969794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3969910Z skip: Need at least 3 CUDA devices (4.022s) 2022-11-23T02:42:49.3969916Z 2022-11-23T02:42:49.3970146Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3970228Z Ran 1 test in 4.022s 2022-11-23T02:42:49.3970233Z 2022-11-23T02:42:49.3970304Z OK (skipped=1) 2022-11-23T02:42:49.3970309Z 2022-11-23T02:42:49.3970402Z Generating XML reports... 2022-11-23T02:42:49.3970757Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123023855.xml 2022-11-23T02:42:49.3971013Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3971427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3971559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3971861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3972007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3972012Z 2022-11-23T02:42:49.3972091Z Running tests... 2022-11-23T02:42:49.3972305Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3972554Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25482 2022-11-23T02:42:49.3972719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25483 2022-11-23T02:42:49.3972884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25484 2022-11-23T02:42:49.3973189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3973317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3973624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3973763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3973941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:42:49.3974234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3974405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3974706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3974853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3975033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3975331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3975452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3975752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3975895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3976068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3976186Z skip: Need at least 3 CUDA devices (4.121s) 2022-11-23T02:42:49.3976191Z 2022-11-23T02:42:49.3976404Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3976484Z Ran 1 test in 4.121s 2022-11-23T02:42:49.3976489Z 2022-11-23T02:42:49.3976599Z OK (skipped=1) 2022-11-23T02:42:49.3976604Z 2022-11-23T02:42:49.3976696Z Generating XML reports... 2022-11-23T02:42:49.3977051Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123023902.xml 2022-11-23T02:42:49.3977300Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3977594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3977727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3978027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3978171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3978176Z 2022-11-23T02:42:49.3978254Z Running tests... 2022-11-23T02:42:49.3978469Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3978736Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: Frequently times out see https://github.com/pytorch/pytorch/issues/58920 (0.000s) 2022-11-23T02:42:49.3978741Z 2022-11-23T02:42:49.3978951Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3979027Z Ran 1 test in 0.000s 2022-11-23T02:42:49.3979032Z 2022-11-23T02:42:49.3979107Z OK (skipped=1) 2022-11-23T02:42:49.3979112Z 2022-11-23T02:42:49.3979199Z Generating XML reports... 2022-11-23T02:42:49.3979556Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123023909.xml 2022-11-23T02:42:49.3979796Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3980096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3980223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3980530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3980679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3980683Z 2022-11-23T02:42:49.3980762Z Running tests... 2022-11-23T02:42:49.3980977Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3981230Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25776 2022-11-23T02:42:49.3981469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25777 2022-11-23T02:42:49.3981632Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25778 2022-11-23T02:42:49.3981931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3982064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3982371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3982510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3982684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3982971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3983101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3983405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3983546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3983757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:42:49.3984047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3984177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3984479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3984608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3984778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3984892Z skip: Need at least 3 CUDA devices (4.028s) 2022-11-23T02:42:49.3984899Z 2022-11-23T02:42:49.3985117Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3985199Z Ran 1 test in 4.029s 2022-11-23T02:42:49.3985204Z 2022-11-23T02:42:49.3985278Z OK (skipped=1) 2022-11-23T02:42:49.3985282Z 2022-11-23T02:42:49.3985375Z Generating XML reports... 2022-11-23T02:42:49.3985734Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123023912.xml 2022-11-23T02:42:49.3985988Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3986292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3986422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3986723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3986870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3986875Z 2022-11-23T02:42:49.3986951Z Running tests... 2022-11-23T02:42:49.3987162Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3987418Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26012 2022-11-23T02:42:49.3987582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26013 2022-11-23T02:42:49.3987750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26014 2022-11-23T02:42:49.3988047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3988184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3988492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3988675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3988838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:42:49.3989133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3989265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3989572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3989718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3989892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3990183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3990314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3990617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3990756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3990961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3991034Z ok (4.123s) 2022-11-23T02:42:49.3991039Z 2022-11-23T02:42:49.3991252Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3991325Z Ran 1 test in 4.124s 2022-11-23T02:42:49.3991330Z 2022-11-23T02:42:49.3991397Z OK 2022-11-23T02:42:49.3991401Z 2022-11-23T02:42:49.3991491Z Generating XML reports... 2022-11-23T02:42:49.3991842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123023919.xml 2022-11-23T02:42:49.3992087Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3992384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3992517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3992827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3992958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3992975Z 2022-11-23T02:42:49.3993040Z Running tests... 2022-11-23T02:42:49.3993255Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3993503Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26248 2022-11-23T02:42:49.3993671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26249 2022-11-23T02:42:49.3993842Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26250 2022-11-23T02:42:49.3994139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3994270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3994581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3994724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3994901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:42:49.3995192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3995324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3995624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3995808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3995975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.3996269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3996398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3996698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3996839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3997009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.3997082Z ok (4.129s) 2022-11-23T02:42:49.3997087Z 2022-11-23T02:42:49.3997288Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3997372Z Ran 1 test in 4.130s 2022-11-23T02:42:49.3997377Z 2022-11-23T02:42:49.3997442Z OK 2022-11-23T02:42:49.3997447Z 2022-11-23T02:42:49.3997534Z Generating XML reports... 2022-11-23T02:42:49.3997928Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123023926.xml 2022-11-23T02:42:49.3998188Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.3998488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.3998613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.3998921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.3999066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.3999073Z 2022-11-23T02:42:49.3999148Z Running tests... 2022-11-23T02:42:49.3999362Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.3999609Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26484 2022-11-23T02:42:49.3999774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26485 2022-11-23T02:42:49.3999940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26486 2022-11-23T02:42:49.4000466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4000598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4000916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4001061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4001241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4001538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4001675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4001972Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4002114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4002288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4002585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4002716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4003021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4003212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4003389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:42:49.4003466Z ok (4.019s) 2022-11-23T02:42:49.4003473Z 2022-11-23T02:42:49.4003698Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4003779Z Ran 1 test in 4.020s 2022-11-23T02:42:49.4003784Z 2022-11-23T02:42:49.4003848Z OK 2022-11-23T02:42:49.4003853Z 2022-11-23T02:42:49.4003942Z Generating XML reports... 2022-11-23T02:42:49.4004305Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123023933.xml 2022-11-23T02:42:49.4004566Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4004867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4005008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4005323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4005513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4005519Z 2022-11-23T02:42:49.4005600Z Running tests... 2022-11-23T02:42:49.4005824Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4006022Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: Test does not pass when run locally (0.001s) 2022-11-23T02:42:49.4006042Z 2022-11-23T02:42:49.4006246Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4006325Z Ran 1 test in 0.001s 2022-11-23T02:42:49.4006329Z 2022-11-23T02:42:49.4006406Z OK (skipped=1) 2022-11-23T02:42:49.4006413Z 2022-11-23T02:42:49.4006502Z Generating XML reports... 2022-11-23T02:42:49.4006860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123023940.xml 2022-11-23T02:42:49.4007112Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4007412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4007546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4007857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4008011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4008016Z 2022-11-23T02:42:49.4008088Z Running tests... 2022-11-23T02:42:49.4008306Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4008551Z test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26778 2022-11-23T02:42:49.4008717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26779 2022-11-23T02:42:49.4008897Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26780 2022-11-23T02:42:49.4009211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4009345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4009664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4009808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4009988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:42:49.4010294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4010474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4010778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4010944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4011123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4011424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4011668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4011971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4012109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4012281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4012392Z skip: Need at least 3 CUDA devices (4.019s) 2022-11-23T02:42:49.4012398Z 2022-11-23T02:42:49.4012611Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4012728Z Ran 1 test in 4.020s 2022-11-23T02:42:49.4012733Z 2022-11-23T02:42:49.4012808Z OK (skipped=1) 2022-11-23T02:42:49.4012812Z 2022-11-23T02:42:49.4012906Z Generating XML reports... 2022-11-23T02:42:49.4013258Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123023942.xml 2022-11-23T02:42:49.4013509Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4013806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4013935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4014243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4014387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4014392Z 2022-11-23T02:42:49.4014471Z Running tests... 2022-11-23T02:42:49.4014690Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4014994Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27014 2022-11-23T02:42:49.4015276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4015407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4015716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4015859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4016031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4016222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4016556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4016626Z ok (3.806s) 2022-11-23T02:42:49.4016631Z 2022-11-23T02:42:49.4016838Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4016912Z Ran 1 test in 3.806s 2022-11-23T02:42:49.4016917Z 2022-11-23T02:42:49.4016980Z OK 2022-11-23T02:42:49.4016988Z 2022-11-23T02:42:49.4017075Z Generating XML reports... 2022-11-23T02:42:49.4017532Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123023949.xml 2022-11-23T02:42:49.4017829Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4018125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4018257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4018564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4018708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4018712Z 2022-11-23T02:42:49.4018791Z Running tests... 2022-11-23T02:42:49.4019004Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4019312Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27139 2022-11-23T02:42:49.4019612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4019742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4020090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4020233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4020407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4020597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4020928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4021546Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:42:49.4021629Z warnings.warn( 2022-11-23T02:42:49.4021699Z ok (3.832s) 2022-11-23T02:42:49.4021704Z 2022-11-23T02:42:49.4021917Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4021997Z Ran 1 test in 3.833s 2022-11-23T02:42:49.4022001Z 2022-11-23T02:42:49.4022063Z OK 2022-11-23T02:42:49.4022068Z 2022-11-23T02:42:49.4022158Z Generating XML reports... 2022-11-23T02:42:49.4022618Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123023956.xml 2022-11-23T02:42:49.4022872Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4023175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4023308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4023614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4023756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4023761Z 2022-11-23T02:42:49.4023840Z Running tests... 2022-11-23T02:42:49.4024054Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4024358Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27262 2022-11-23T02:42:49.4024657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4024788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4025096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4025370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4025554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4025739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4026065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4026133Z ok (3.805s) 2022-11-23T02:42:49.4026138Z 2022-11-23T02:42:49.4026356Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4026435Z Ran 1 test in 3.805s 2022-11-23T02:42:49.4026440Z 2022-11-23T02:42:49.4026502Z OK 2022-11-23T02:42:49.4026507Z 2022-11-23T02:42:49.4026595Z Generating XML reports... 2022-11-23T02:42:49.4027054Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024003.xml 2022-11-23T02:42:49.4027308Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4027605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4027779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4028089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4028229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4028234Z 2022-11-23T02:42:49.4028307Z Running tests... 2022-11-23T02:42:49.4028523Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4028837Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27385 2022-11-23T02:42:49.4029138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4029269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4029579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4029728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4029903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4030080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4030407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4030480Z ok (3.807s) 2022-11-23T02:42:49.4030485Z 2022-11-23T02:42:49.4030703Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4030784Z Ran 1 test in 3.808s 2022-11-23T02:42:49.4030789Z 2022-11-23T02:42:49.4030851Z OK 2022-11-23T02:42:49.4030856Z 2022-11-23T02:42:49.4030940Z Generating XML reports... 2022-11-23T02:42:49.4031802Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123024010.xml 2022-11-23T02:42:49.4032049Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4032346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4032478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4032777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4032924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4032971Z 2022-11-23T02:42:49.4033046Z Running tests... 2022-11-23T02:42:49.4033257Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4033457Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: GPUs are available, skipping test (0.000s) 2022-11-23T02:42:49.4033465Z 2022-11-23T02:42:49.4033676Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4033757Z Ran 1 test in 0.001s 2022-11-23T02:42:49.4033762Z 2022-11-23T02:42:49.4033835Z OK (skipped=1) 2022-11-23T02:42:49.4033839Z 2022-11-23T02:42:49.4033928Z Generating XML reports... 2022-11-23T02:42:49.4034300Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123024016.xml 2022-11-23T02:42:49.4034554Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4034854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4034989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4035296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4035469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4035474Z 2022-11-23T02:42:49.4035550Z Running tests... 2022-11-23T02:42:49.4035766Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4036014Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27568 2022-11-23T02:42:49.4036180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27569 2022-11-23T02:42:49.4036476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4036608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4036915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4037055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4037233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4037425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4037721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4037840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4038146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4038291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4038468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4038656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4038979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4039303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4039375Z ok (3.730s) 2022-11-23T02:42:49.4039380Z 2022-11-23T02:42:49.4039594Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4039668Z Ran 1 test in 3.731s 2022-11-23T02:42:49.4039673Z 2022-11-23T02:42:49.4039736Z OK 2022-11-23T02:42:49.4039740Z 2022-11-23T02:42:49.4039829Z Generating XML reports... 2022-11-23T02:42:49.4040255Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024019.xml 2022-11-23T02:42:49.4040728Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4041040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4041176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4041488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4041639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4041644Z 2022-11-23T02:42:49.4041720Z Running tests... 2022-11-23T02:42:49.4041940Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4042192Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27749 2022-11-23T02:42:49.4042371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27750 2022-11-23T02:42:49.4042678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4042843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4043166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4043311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4043491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4043685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4043992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4044130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4044443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4044592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4044775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4045144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4045478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4045803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4045879Z ok (4.215s) 2022-11-23T02:42:49.4045885Z 2022-11-23T02:42:49.4046105Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4046190Z Ran 1 test in 4.215s 2022-11-23T02:42:49.4046195Z 2022-11-23T02:42:49.4046258Z OK 2022-11-23T02:42:49.4046263Z 2022-11-23T02:42:49.4046355Z Generating XML reports... 2022-11-23T02:42:49.4046720Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024026.xml 2022-11-23T02:42:49.4046976Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4047281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4047416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4047717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4047862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4047907Z 2022-11-23T02:42:49.4047994Z Running tests... 2022-11-23T02:42:49.4048215Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4048459Z test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27937 2022-11-23T02:42:49.4048633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27938 2022-11-23T02:42:49.4048942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4049074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4049390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4049536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4049716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4049913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4050217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4050384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4050703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4050849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4051024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4051321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4051644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4051968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4052042Z ok (4.125s) 2022-11-23T02:42:49.4052046Z 2022-11-23T02:42:49.4052268Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4052340Z Ran 1 test in 4.126s 2022-11-23T02:42:49.4052359Z 2022-11-23T02:42:49.4052412Z OK 2022-11-23T02:42:49.4052417Z 2022-11-23T02:42:49.4052502Z Generating XML reports... 2022-11-23T02:42:49.4052858Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024033.xml 2022-11-23T02:42:49.4053281Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4053606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4053738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4054047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4054207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4054213Z 2022-11-23T02:42:49.4054290Z Running tests... 2022-11-23T02:42:49.4054557Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4054798Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28129 2022-11-23T02:42:49.4054966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28130 2022-11-23T02:42:49.4055268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4055401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4055720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4055909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4056093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4056357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4056665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4056802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4057119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4057268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4057438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4057632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4057968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4058333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4058404Z ok (4.617s) 2022-11-23T02:42:49.4058409Z 2022-11-23T02:42:49.4058628Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4058705Z Ran 1 test in 4.617s 2022-11-23T02:42:49.4058711Z 2022-11-23T02:42:49.4058780Z OK 2022-11-23T02:42:49.4058785Z 2022-11-23T02:42:49.4058879Z Generating XML reports... 2022-11-23T02:42:49.4059237Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024040.xml 2022-11-23T02:42:49.4059496Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4059804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4059935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4060252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4060404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4060409Z 2022-11-23T02:42:49.4060488Z Running tests... 2022-11-23T02:42:49.4060708Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4060946Z test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28321 2022-11-23T02:42:49.4061120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28322 2022-11-23T02:42:49.4061429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4061568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4061870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4062020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4062195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4062384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4062686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4062819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4063135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4063319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4063497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4063693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4064029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4064359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4064430Z ok (4.016s) 2022-11-23T02:42:49.4064435Z 2022-11-23T02:42:49.4064652Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4064736Z Ran 1 test in 4.016s 2022-11-23T02:42:49.4064741Z 2022-11-23T02:42:49.4064804Z OK 2022-11-23T02:42:49.4064811Z 2022-11-23T02:42:49.4065006Z Generating XML reports... 2022-11-23T02:42:49.4065363Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024047.xml 2022-11-23T02:42:49.4065612Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4065946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4066082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4066494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4066619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4066638Z 2022-11-23T02:42:49.4066700Z Running tests... 2022-11-23T02:42:49.4066903Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4067126Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28509 2022-11-23T02:42:49.4067285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28510 2022-11-23T02:42:49.4067569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4067691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4067976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4068112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4068273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4068449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4068730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4068857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4069147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4069283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4069448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4069623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4069926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4070227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4070339Z ok (3.998s) 2022-11-23T02:42:49.4070343Z 2022-11-23T02:42:49.4070548Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4070628Z Ran 1 test in 3.999s 2022-11-23T02:42:49.4070632Z 2022-11-23T02:42:49.4070681Z OK 2022-11-23T02:42:49.4070695Z 2022-11-23T02:42:49.4070767Z Generating XML reports... 2022-11-23T02:42:49.4071106Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024054.xml 2022-11-23T02:42:49.4071343Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4071624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4071749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4072035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4072171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4072175Z 2022-11-23T02:42:49.4072247Z Running tests... 2022-11-23T02:42:49.4072447Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4072702Z test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28697 2022-11-23T02:42:49.4072863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28698 2022-11-23T02:42:49.4073140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4073261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4073549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4073680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4073845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4074026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4074306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4074426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4074712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4074849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4075003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4075176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4075478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4075781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4075848Z ok (4.022s) 2022-11-23T02:42:49.4075852Z 2022-11-23T02:42:49.4076055Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4076129Z Ran 1 test in 4.023s 2022-11-23T02:42:49.4076134Z 2022-11-23T02:42:49.4076195Z OK 2022-11-23T02:42:49.4076199Z 2022-11-23T02:42:49.4076284Z Generating XML reports... 2022-11-23T02:42:49.4076613Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024101.xml 2022-11-23T02:42:49.4076849Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4077304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4077489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4077795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4077934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4077941Z 2022-11-23T02:42:49.4078017Z Running tests... 2022-11-23T02:42:49.4078228Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4078466Z test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28885 2022-11-23T02:42:49.4078631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28886 2022-11-23T02:42:49.4078933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4079064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4079372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4079506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4079713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4079900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4080256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4080552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4080866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4081014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4081197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4081390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4081724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4082058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4082136Z ok (3.711s) 2022-11-23T02:42:49.4082141Z 2022-11-23T02:42:49.4082363Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4082445Z Ran 1 test in 3.712s 2022-11-23T02:42:49.4082450Z 2022-11-23T02:42:49.4082516Z OK 2022-11-23T02:42:49.4082520Z 2022-11-23T02:42:49.4082610Z Generating XML reports... 2022-11-23T02:42:49.4082975Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024108.xml 2022-11-23T02:42:49.4083234Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4083543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4083680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4083993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4084137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4084142Z 2022-11-23T02:42:49.4084218Z Running tests... 2022-11-23T02:42:49.4084428Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4084665Z test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29066 2022-11-23T02:42:49.4084831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29067 2022-11-23T02:42:49.4085183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4085318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4085632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4085775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4085953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4086147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4086449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4086585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4086901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4087046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4087223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4087458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4087797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4088127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4088198Z ok (4.113s) 2022-11-23T02:42:49.4088203Z 2022-11-23T02:42:49.4088426Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4088505Z Ran 1 test in 4.114s 2022-11-23T02:42:49.4088512Z 2022-11-23T02:42:49.4088576Z OK 2022-11-23T02:42:49.4088581Z 2022-11-23T02:42:49.4088660Z Generating XML reports... 2022-11-23T02:42:49.4089017Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024115.xml 2022-11-23T02:42:49.4089275Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4089581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4089715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4090025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4090170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4090174Z 2022-11-23T02:42:49.4090249Z Running tests... 2022-11-23T02:42:49.4090466Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4090711Z test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29254 2022-11-23T02:42:49.4090880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29255 2022-11-23T02:42:49.4091185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4091421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4091725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4091867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4092044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4092332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4092655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4092782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4093074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4093208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4093372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4093539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4093845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4094148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4094216Z ok (7.828s) 2022-11-23T02:42:49.4094220Z 2022-11-23T02:42:49.4094422Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4094495Z Ran 1 test in 7.828s 2022-11-23T02:42:49.4094499Z 2022-11-23T02:42:49.4094559Z OK 2022-11-23T02:42:49.4094563Z 2022-11-23T02:42:49.4094680Z Generating XML reports... 2022-11-23T02:42:49.4095015Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024122.xml 2022-11-23T02:42:49.4095251Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4095529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4095651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4095933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4096067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4096072Z 2022-11-23T02:42:49.4096143Z Running tests... 2022-11-23T02:42:49.4096348Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4096581Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29442 2022-11-23T02:42:49.4096741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29443 2022-11-23T02:42:49.4097025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4097153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4097445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4097586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4097741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4097922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4098208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4098340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4098635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4098773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4098941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4099119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4099460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4099760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4099833Z ok (4.024s) 2022-11-23T02:42:49.4099838Z 2022-11-23T02:42:49.4100042Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4100124Z Ran 1 test in 4.024s 2022-11-23T02:42:49.4100129Z 2022-11-23T02:42:49.4100188Z OK 2022-11-23T02:42:49.4100193Z 2022-11-23T02:42:49.4100279Z Generating XML reports... 2022-11-23T02:42:49.4100610Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024133.xml 2022-11-23T02:42:49.4100846Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4101128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4101255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4101542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4101711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4101716Z 2022-11-23T02:42:49.4101791Z Running tests... 2022-11-23T02:42:49.4101983Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4102203Z test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29628 2022-11-23T02:42:49.4102364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29629 2022-11-23T02:42:49.4102645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4102772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4103066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4103204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4103374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4103553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4103834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4103963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4104252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4104393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4104561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4104737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4105052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4105355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4105425Z ok (4.313s) 2022-11-23T02:42:49.4105430Z 2022-11-23T02:42:49.4105636Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4105712Z Ran 1 test in 4.313s 2022-11-23T02:42:49.4105717Z 2022-11-23T02:42:49.4105777Z OK 2022-11-23T02:42:49.4105781Z 2022-11-23T02:42:49.4105866Z Generating XML reports... 2022-11-23T02:42:49.4106222Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024140.xml 2022-11-23T02:42:49.4106458Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4106738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4106861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4107148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4107283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4107287Z 2022-11-23T02:42:49.4107359Z Running tests... 2022-11-23T02:42:49.4107559Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4107799Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29820 2022-11-23T02:42:49.4107959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29821 2022-11-23T02:42:49.4108243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4108408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4108698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4108836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4108996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4109177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4109457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4109582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4109873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4110013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4110181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4110356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4110652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4110956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4111028Z ok (3.611s) 2022-11-23T02:42:49.4111033Z 2022-11-23T02:42:49.4111236Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4111309Z Ran 1 test in 3.612s 2022-11-23T02:42:49.4111314Z 2022-11-23T02:42:49.4111372Z OK 2022-11-23T02:42:49.4111377Z 2022-11-23T02:42:49.4111461Z Generating XML reports... 2022-11-23T02:42:49.4111794Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024147.xml 2022-11-23T02:42:49.4112032Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4112323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4112447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4112731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4112864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4112904Z 2022-11-23T02:42:49.4112974Z Running tests... 2022-11-23T02:42:49.4113179Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4113413Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30001 2022-11-23T02:42:49.4113577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30002 2022-11-23T02:42:49.4113861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4113987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4114278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4114412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4114579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4114749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4115036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4115192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4115488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4115623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4115791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4115968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4116456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4116776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4116850Z ok (3.922s) 2022-11-23T02:42:49.4116855Z 2022-11-23T02:42:49.4117066Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4117149Z Ran 1 test in 3.922s 2022-11-23T02:42:49.4117154Z 2022-11-23T02:42:49.4117217Z OK 2022-11-23T02:42:49.4117221Z 2022-11-23T02:42:49.4117310Z Generating XML reports... 2022-11-23T02:42:49.4117660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024153.xml 2022-11-23T02:42:49.4117909Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4118202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4118333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4118641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4118783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4118789Z 2022-11-23T02:42:49.4118863Z Running tests... 2022-11-23T02:42:49.4119080Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4119310Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30189 2022-11-23T02:42:49.4119479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30190 2022-11-23T02:42:49.4119782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4119912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4120276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4120637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4120819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4121015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4121329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4121464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4121777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4121925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4122109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4122304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4122639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4123016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4123089Z ok (4.120s) 2022-11-23T02:42:49.4123095Z 2022-11-23T02:42:49.4123312Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4123393Z Ran 1 test in 4.121s 2022-11-23T02:42:49.4123398Z 2022-11-23T02:42:49.4123462Z OK 2022-11-23T02:42:49.4123466Z 2022-11-23T02:42:49.4123559Z Generating XML reports... 2022-11-23T02:42:49.4123920Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024200.xml 2022-11-23T02:42:49.4124169Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4124473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4124604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4124915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4125060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4125065Z 2022-11-23T02:42:49.4125144Z Running tests... 2022-11-23T02:42:49.4125366Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4125614Z test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30381 2022-11-23T02:42:49.4125788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30382 2022-11-23T02:42:49.4126099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4126232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4126548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4126693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4126872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4127062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4127366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4127502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4127821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4128006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4128183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4128375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4128710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4129035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4129098Z ok (3.623s) 2022-11-23T02:42:49.4129103Z 2022-11-23T02:42:49.4129325Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4129403Z Ran 1 test in 3.624s 2022-11-23T02:42:49.4129408Z 2022-11-23T02:42:49.4129472Z OK 2022-11-23T02:42:49.4129478Z 2022-11-23T02:42:49.4129564Z Generating XML reports... 2022-11-23T02:42:49.4129924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024207.xml 2022-11-23T02:42:49.4130179Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4130530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4130665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4130981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4131125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4131131Z 2022-11-23T02:42:49.4131317Z Running tests... 2022-11-23T02:42:49.4131539Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4131780Z test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30562 2022-11-23T02:42:49.4132056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30563 2022-11-23T02:42:49.4132342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4132470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4132761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4132900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4133064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4133240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4133509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4133634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4133926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4134065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4134231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4134409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4134713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4135018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4135130Z ok (4.126s) 2022-11-23T02:42:49.4135135Z 2022-11-23T02:42:49.4135336Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4135409Z Ran 1 test in 4.127s 2022-11-23T02:42:49.4135414Z 2022-11-23T02:42:49.4135473Z OK 2022-11-23T02:42:49.4135477Z 2022-11-23T02:42:49.4135557Z Generating XML reports... 2022-11-23T02:42:49.4135888Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024214.xml 2022-11-23T02:42:49.4136123Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4136404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4136524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4136812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4136947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4136951Z 2022-11-23T02:42:49.4137025Z Running tests... 2022-11-23T02:42:49.4137227Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4137484Z test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30750 2022-11-23T02:42:49.4137632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30751 2022-11-23T02:42:49.4137913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4138036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4138325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4138460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4138627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4138803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4139089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4139216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4139506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4139641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4139809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4139985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4140293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4140598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4140665Z ok (7.827s) 2022-11-23T02:42:49.4140670Z 2022-11-23T02:42:49.4140873Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4140947Z Ran 1 test in 7.828s 2022-11-23T02:42:49.4140952Z 2022-11-23T02:42:49.4141009Z OK 2022-11-23T02:42:49.4141013Z 2022-11-23T02:42:49.4141094Z Generating XML reports... 2022-11-23T02:42:49.4141423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024221.xml 2022-11-23T02:42:49.4141655Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4141925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4142084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4142372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4142507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4142512Z 2022-11-23T02:42:49.4142582Z Running tests... 2022-11-23T02:42:49.4142785Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4143006Z test_send_recv (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30938 2022-11-23T02:42:49.4143163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30939 2022-11-23T02:42:49.4143444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4143568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4143858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4143993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4144193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:42:49.4144371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:42:49.4144652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4144775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4145063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4145201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4145370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:42:49.4145548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4145857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4146162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:42:49.4146220Z ok (4.616s) 2022-11-23T02:42:49.4146233Z 2022-11-23T02:42:49.4146425Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4146498Z Ran 1 test in 4.616s 2022-11-23T02:42:49.4146503Z 2022-11-23T02:42:49.4146565Z OK 2022-11-23T02:42:49.4146570Z 2022-11-23T02:42:49.4146650Z Generating XML reports... 2022-11-23T02:42:49.4146978Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123024232.xml 2022-11-23T02:42:49.4147215Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4147496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4147621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4147908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4148045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4148050Z 2022-11-23T02:42:49.4148124Z Running tests... 2022-11-23T02:42:49.4148324Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4148557Z test_common_errors (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4149052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4149227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4149535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4149710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4150008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4150179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4150482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4150553Z ok (0.504s) 2022-11-23T02:42:49.4150557Z 2022-11-23T02:42:49.4150759Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4150823Z Ran 1 test in 0.505s 2022-11-23T02:42:49.4150839Z 2022-11-23T02:42:49.4150889Z OK 2022-11-23T02:42:49.4150893Z 2022-11-23T02:42:49.4150983Z Generating XML reports... 2022-11-23T02:42:49.4151339Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123024239.xml 2022-11-23T02:42:49.4151582Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T02:42:49.4151866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:42:49.4151989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:42:49.4152276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:42:49.4152419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:42:49.4152423Z 2022-11-23T02:42:49.4152497Z Running tests... 2022-11-23T02:42:49.4152699Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4152937Z test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4153244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4153417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:42:49.4153721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:42:49.4153790Z ok (3.537s) 2022-11-23T02:42:49.4153795Z 2022-11-23T02:42:49.4153997Z ---------------------------------------------------------------------- 2022-11-23T02:42:49.4154076Z Ran 1 test in 3.538s 2022-11-23T02:42:49.4154081Z 2022-11-23T02:42:49.4154141Z OK 2022-11-23T02:42:49.4154145Z 2022-11-23T02:42:49.4154230Z Generating XML reports... 2022-11-23T02:42:49.4154533Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123024243.xml 2022-11-23T02:42:49.4154540Z 2022-11-23T02:42:49.4154908Z ##[endgroup] 2022-11-23T02:42:49.4155232Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_nccl_4y72wl21) 2022-11-23T02:42:49.4155266Z 2022-11-23T02:42:49.4155460Z Running distributed/test_c10d_error_logger ... [2022-11-23 02:42:49.269904] 2022-11-23T02:42:49.4155885Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_error_logger.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:42:49.270754] 2022-11-23T02:42:52.7416993Z 2022-11-23T02:42:52.7417934Z Expand the folded group to see the log file of distributed/test_c10d_error_logger 2022-11-23T02:42:52.7421215Z ##[group]PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_error_logger_5m5x1e1g) 2022-11-23T02:42:52.7423793Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_error_logger 2022-11-23T02:42:52.7424616Z 2022-11-23T02:42:52.7424958Z Running tests... 2022-11-23T02:42:52.7426675Z ---------------------------------------------------------------------- 2022-11-23T02:42:52.7429001Z test_exception_handler_with_dist (__main__.C10dErrorLoggerTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.487s) 2022-11-23T02:42:52.7431715Z test_get_or_create_logger (__main__.C10dErrorLoggerTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:42:52.7432762Z 2022-11-23T02:42:52.7457376Z ---------------------------------------------------------------------- 2022-11-23T02:42:52.7458375Z Ran 2 tests in 0.488s 2022-11-23T02:42:52.7458787Z 2022-11-23T02:42:52.7459029Z OK (skipped=2) 2022-11-23T02:42:52.7459409Z 2022-11-23T02:42:52.7459707Z Generating XML reports... 2022-11-23T02:42:52.7461879Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_error_logger/TEST-C10dErrorLoggerTest-20221123024250.xml 2022-11-23T02:42:52.7462886Z 2022-11-23T02:42:52.7463725Z ##[endgroup] 2022-11-23T02:42:52.7465633Z FINISHED PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_error_logger_5m5x1e1g) 2022-11-23T02:42:52.7466555Z 2022-11-23T02:42:52.7467333Z Running distributed/pipeline/sync/test_worker ... [2022-11-23 02:42:52.742298] 2022-11-23T02:42:52.7469039Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-11-23 02:42:52.743132] 2022-11-23T02:42:56.2099940Z 2022-11-23T02:42:56.2101188Z Expand the folded group to see the log file of distributed/pipeline/sync/test_worker 2022-11-23T02:42:56.2103577Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_worker_3j5x__hr) 2022-11-23T02:42:56.2104992Z ============================= test session starts ============================== 2022-11-23T02:42:56.2106679Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:42:56.2107602Z cachedir: .pytest_cache 2022-11-23T02:42:56.2109202Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:42:56.2110341Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:42:56.2111175Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:42:56.2112718Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:42:56.2113796Z collecting ... collected 6 items 2022-11-23T02:42:56.2116178Z Running 6 items in this shard: test/distributed/pipeline/sync/test_worker.py::test_compute_multithreading, test/distributed/pipeline/sync/test_worker.py::test_compute_success, test/distributed/pipeline/sync/test_worker.py::test_compute_exception, test/distributed/pipeline/sync/test_worker.py::test_grad_mode[True], test/distributed/pipeline/sync/test_worker.py::test_grad_mode[False], test/distributed/pipeline/sync/test_worker.py::test_worker_per_device 2022-11-23T02:42:56.2118092Z 2022-11-23T02:42:56.2118655Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%] 2022-11-23T02:42:56.2119851Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED [ 33%] 2022-11-23T02:42:56.2121260Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED [ 50%] 2022-11-23T02:42:56.2122406Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED [ 66%] 2022-11-23T02:42:56.2123982Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED [ 83%] 2022-11-23T02:42:56.2125125Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED [100%] 2022-11-23T02:42:56.2125756Z 2022-11-23T02:42:56.2126145Z ============================== 6 passed in 0.04s =============================== 2022-11-23T02:42:56.2126638Z 2022-11-23T02:42:56.2127383Z ##[endgroup] 2022-11-23T02:42:56.2129143Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_worker_3j5x__hr) 2022-11-23T02:42:56.2130144Z 2022-11-23T02:42:56.2130941Z Running distributed/pipeline/sync/test_stream ... [2022-11-23 02:42:56.210560] 2022-11-23T02:42:56.2132632Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-11-23 02:42:56.211380] 2022-11-23T02:43:00.2764889Z 2022-11-23T02:43:00.2766308Z Expand the folded group to see the log file of distributed/pipeline/sync/test_stream 2022-11-23T02:43:00.2770016Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_stream_yt05b0o2) 2022-11-23T02:43:00.2771424Z ============================= test session starts ============================== 2022-11-23T02:43:00.2773510Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:00.2774460Z cachedir: .pytest_cache 2022-11-23T02:43:00.2776457Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:00.2777608Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:00.2778445Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:00.2780013Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:00.2781051Z collecting ... collected 19 items 2022-11-23T02:43:00.2788482Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view 2022-11-23T02:43:00.2795158Z 2022-11-23T02:43:00.2795777Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [ 5%] 2022-11-23T02:43:00.2797067Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda PASSED [ 10%] 2022-11-23T02:43:00.2798375Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%] 2022-11-23T02:43:00.2799937Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda PASSED [ 21%] 2022-11-23T02:43:00.2801480Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%] 2022-11-23T02:43:00.2802833Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda PASSED [ 31%] 2022-11-23T02:43:00.2804134Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%] 2022-11-23T02:43:00.2805390Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda PASSED [ 42%] 2022-11-23T02:43:00.2806642Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%] 2022-11-23T02:43:00.2807887Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda PASSED [ 52%] 2022-11-23T02:43:00.2809108Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%] 2022-11-23T02:43:00.2810348Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda PASSED [ 63%] 2022-11-23T02:43:00.2811644Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%] 2022-11-23T02:43:00.2813074Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda PASSED [ 73%] 2022-11-23T02:43:00.2814370Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu PASSED [ 78%] 2022-11-23T02:43:00.2815676Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda PASSED [ 84%] 2022-11-23T02:43:00.2816992Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%] 2022-11-23T02:43:00.2818297Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda PASSED [ 94%] 2022-11-23T02:43:00.2819634Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view PASSED [100%] 2022-11-23T02:43:00.2820386Z 2022-11-23T02:43:00.2820787Z ============================== 19 passed in 1.08s ============================== 2022-11-23T02:43:00.2821274Z 2022-11-23T02:43:00.2822067Z ##[endgroup] 2022-11-23T02:43:00.2823895Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_stream_yt05b0o2) 2022-11-23T02:43:00.2824910Z 2022-11-23T02:43:00.2825696Z Running distributed/pipeline/sync/test_pipe ... [2022-11-23 02:43:00.277169] 2022-11-23T02:43:00.2827373Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-11-23 02:43:00.277983] 2022-11-23T02:43:09.9849447Z 2022-11-23T02:43:09.9850861Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipe 2022-11-23T02:43:09.9853362Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_pipe_62sl7lwi) 2022-11-23T02:43:09.9854790Z ============================= test session starts ============================== 2022-11-23T02:43:09.9856422Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:09.9857430Z cachedir: .pytest_cache 2022-11-23T02:43:09.9859010Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:09.9860178Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:09.9861021Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:09.9862590Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:09.9863642Z collecting ... collected 56 items 2022-11-23T02:43:09.9880867Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper 2022-11-23T02:43:09.9896580Z 2022-11-23T02:43:09.9897153Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED [ 1%] 2022-11-23T02:43:09.9898317Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED [ 3%] 2022-11-23T02:43:09.9899567Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED [ 5%] 2022-11-23T02:43:09.9900666Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED [ 7%] 2022-11-23T02:43:09.9901800Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED [ 8%] 2022-11-23T02:43:09.9902947Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%] 2022-11-23T02:43:09.9904089Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED [ 12%] 2022-11-23T02:43:09.9905188Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED [ 14%] 2022-11-23T02:43:09.9906361Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%] 2022-11-23T02:43:09.9907572Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%] 2022-11-23T02:43:09.9908747Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED [ 19%] 2022-11-23T02:43:09.9909942Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%] 2022-11-23T02:43:09.9911085Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED [ 23%] 2022-11-23T02:43:09.9912287Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED [ 25%] 2022-11-23T02:43:09.9913459Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%] 2022-11-23T02:43:09.9914627Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED [ 28%] 2022-11-23T02:43:09.9915732Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED [ 30%] 2022-11-23T02:43:09.9916822Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%] 2022-11-23T02:43:09.9917947Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED [ 33%] 2022-11-23T02:43:09.9919066Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED [ 35%] 2022-11-23T02:43:09.9920351Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED [ 37%] 2022-11-23T02:43:09.9921522Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED [ 39%] 2022-11-23T02:43:09.9922666Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%] 2022-11-23T02:43:09.9923850Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%] 2022-11-23T02:43:09.9924924Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%] 2022-11-23T02:43:09.9925401Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%] 2022-11-23T02:43:09.9925871Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%] 2022-11-23T02:43:09.9926349Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%] 2022-11-23T02:43:09.9926830Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%] 2022-11-23T02:43:09.9927304Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%] 2022-11-23T02:43:09.9927785Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%] 2022-11-23T02:43:09.9928237Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED [ 57%] 2022-11-23T02:43:09.9928685Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED [ 58%] 2022-11-23T02:43:09.9929143Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%] 2022-11-23T02:43:09.9929611Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%] 2022-11-23T02:43:09.9930090Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%] 2022-11-23T02:43:09.9930583Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%] 2022-11-23T02:43:09.9931079Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%] 2022-11-23T02:43:09.9931629Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%] 2022-11-23T02:43:09.9932099Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED [ 71%] 2022-11-23T02:43:09.9932541Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED [ 73%] 2022-11-23T02:43:09.9932991Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions PASSED [ 75%] 2022-11-23T02:43:09.9933435Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED [ 76%] 2022-11-23T02:43:09.9933872Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED [ 78%] 2022-11-23T02:43:09.9934307Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED [ 80%] 2022-11-23T02:43:09.9934755Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%] 2022-11-23T02:43:09.9935244Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%] 2022-11-23T02:43:09.9935749Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device PASSED [ 85%] 2022-11-23T02:43:09.9936231Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules PASSED [ 87%] 2022-11-23T02:43:09.9936794Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%] 2022-11-23T02:43:09.9937300Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED [ 91%] 2022-11-23T02:43:09.9937764Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] PASSED [ 92%] 2022-11-23T02:43:09.9938217Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] PASSED [ 94%] 2022-11-23T02:43:09.9938693Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] PASSED [ 96%] 2022-11-23T02:43:09.9939168Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device PASSED [ 98%] 2022-11-23T02:43:09.9939624Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper PASSED [100%] 2022-11-23T02:43:09.9939878Z 2022-11-23T02:43:09.9940038Z ============================== 56 passed in 6.30s ============================== 2022-11-23T02:43:09.9940231Z 2022-11-23T02:43:09.9940566Z ##[endgroup] 2022-11-23T02:43:09.9941302Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_pipe_62sl7lwi) 2022-11-23T02:43:09.9941693Z 2022-11-23T02:43:09.9942021Z Running distributed/pipeline/sync/test_microbatch ... [2022-11-23 02:43:09.985906] 2022-11-23T02:43:09.9942694Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-11-23 02:43:09.986739] 2022-11-23T02:43:13.0290349Z 2022-11-23T02:43:13.0291953Z Expand the folded group to see the log file of distributed/pipeline/sync/test_microbatch 2022-11-23T02:43:13.0294772Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_microbatch_ia8f5nwd) 2022-11-23T02:43:13.0296243Z ============================= test session starts ============================== 2022-11-23T02:43:13.0297904Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:13.0298827Z cachedir: .pytest_cache 2022-11-23T02:43:13.0300467Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:13.0301629Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:13.0302477Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:13.0304056Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:13.0305097Z collecting ... collected 10 items 2022-11-23T02:43:13.0308657Z Running 10 items in this shard: test/distributed/pipeline/sync/test_microbatch.py::test_batch_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_call, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice, test/distributed/pipeline/sync/test_microbatch.py::test_check, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tensors, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tuples, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors 2022-11-23T02:43:13.0312129Z 2022-11-23T02:43:13.0312698Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED [ 10%] 2022-11-23T02:43:13.0313892Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%] 2022-11-23T02:43:13.0315061Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED [ 30%] 2022-11-23T02:43:13.0316224Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%] 2022-11-23T02:43:13.0317455Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%] 2022-11-23T02:43:13.0318622Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED [ 60%] 2022-11-23T02:43:13.0319935Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%] 2022-11-23T02:43:13.0321390Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED [ 80%] 2022-11-23T02:43:13.0322542Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%] 2022-11-23T02:43:13.0323764Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%] 2022-11-23T02:43:13.0324455Z 2022-11-23T02:43:13.0324826Z ============================== 10 passed in 0.04s ============================== 2022-11-23T02:43:13.0325316Z 2022-11-23T02:43:13.0326084Z ##[endgroup] 2022-11-23T02:43:13.0327949Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_microbatch_ia8f5nwd) 2022-11-23T02:43:13.0328997Z 2022-11-23T02:43:13.0329806Z Running distributed/pipeline/sync/test_dependency ... [2022-11-23 02:43:13.029232] 2022-11-23T02:43:13.0331560Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-11-23 02:43:13.029864] 2022-11-23T02:43:16.3991201Z 2022-11-23T02:43:16.3992544Z Expand the folded group to see the log file of distributed/pipeline/sync/test_dependency 2022-11-23T02:43:16.3994990Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_dependency_ibg7dxun) 2022-11-23T02:43:16.3996436Z ============================= test session starts ============================== 2022-11-23T02:43:16.3998341Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:16.3999270Z cachedir: .pytest_cache 2022-11-23T02:43:16.4001065Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:16.4002228Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:16.4003068Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:16.4004665Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:16.4005856Z collecting ... collected 6 items 2022-11-23T02:43:16.4008375Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad 2022-11-23T02:43:16.4010942Z 2022-11-23T02:43:16.4011505Z distributed/pipeline/sync/test_dependency.py::test_fork_join PASSED [ 16%] 2022-11-23T02:43:16.4012715Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%] 2022-11-23T02:43:16.4013941Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%] 2022-11-23T02:43:16.4015112Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED [ 66%] 2022-11-23T02:43:16.4016314Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%] 2022-11-23T02:43:16.4017612Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%] 2022-11-23T02:43:16.4018300Z 2022-11-23T02:43:16.4018697Z ============================== 6 passed in 0.31s =============================== 2022-11-23T02:43:16.4019179Z 2022-11-23T02:43:16.4019947Z ##[endgroup] 2022-11-23T02:43:16.4021814Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_dependency_ibg7dxun) 2022-11-23T02:43:16.4022880Z 2022-11-23T02:43:16.4023666Z Running distributed/pipeline/sync/test_copy ... [2022-11-23 02:43:16.399621] 2022-11-23T02:43:16.4025533Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-11-23 02:43:16.400435] 2022-11-23T02:43:26.3705071Z 2022-11-23T02:43:26.3706676Z Expand the folded group to see the log file of distributed/pipeline/sync/test_copy 2022-11-23T02:43:26.3709703Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_copy_8zcnlcws) 2022-11-23T02:43:26.3710986Z ============================= test session starts ============================== 2022-11-23T02:43:26.3712504Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:26.3713396Z cachedir: .pytest_cache 2022-11-23T02:43:26.3714838Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:26.3715891Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:26.3716657Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:26.3718090Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:26.3719055Z collecting ... collected 5 items 2022-11-23T02:43:26.3721417Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors 2022-11-23T02:43:26.3723107Z 2022-11-23T02:43:26.3723655Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED [ 20%] 2022-11-23T02:43:26.3724824Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda PASSED [ 40%] 2022-11-23T02:43:26.3725962Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu PASSED [ 60%] 2022-11-23T02:43:26.3727064Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda PASSED [ 80%] 2022-11-23T02:43:26.3728214Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%] 2022-11-23T02:43:26.3728866Z 2022-11-23T02:43:26.3729260Z ============================== 5 passed in 7.00s =============================== 2022-11-23T02:43:26.3729745Z 2022-11-23T02:43:26.3730501Z ##[endgroup] 2022-11-23T02:43:26.3732248Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_copy_8zcnlcws) 2022-11-23T02:43:26.3733240Z 2022-11-23T02:43:26.3734005Z Running distributed/pipeline/sync/test_bugs ... [2022-11-23 02:43:26.370514] 2022-11-23T02:43:26.3736174Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-11-23 02:43:26.371034] 2022-11-23T02:43:31.6021974Z 2022-11-23T02:43:31.6023156Z Expand the folded group to see the log file of distributed/pipeline/sync/test_bugs 2022-11-23T02:43:31.6024715Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_bugs_xqt3kb_v) 2022-11-23T02:43:31.6025570Z ============================= test session starts ============================== 2022-11-23T02:43:31.6026551Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:31.6027045Z cachedir: .pytest_cache 2022-11-23T02:43:31.6027897Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:31.6028495Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:31.6028951Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:31.6029800Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:31.6030369Z collecting ... collected 4 items 2022-11-23T02:43:31.6031713Z Running 4 items in this shard: test/distributed/pipeline/sync/test_bugs.py::test_python_autograd_function, test/distributed/pipeline/sync/test_bugs.py::test_exception_no_hang, test/distributed/pipeline/sync/test_bugs.py::test_tuple_wait, test/distributed/pipeline/sync/test_bugs.py::test_parallel_randoms 2022-11-23T02:43:31.6032647Z 2022-11-23T02:43:31.6032958Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%] 2022-11-23T02:43:31.6033586Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED [ 50%] 2022-11-23T02:43:31.6034182Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait PASSED [ 75%] 2022-11-23T02:43:31.6034755Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED [100%] 2022-11-23T02:43:31.6035108Z 2022-11-23T02:43:31.6035318Z ============================== 4 passed in 2.07s =============================== 2022-11-23T02:43:31.6035583Z 2022-11-23T02:43:31.6036001Z ##[endgroup] 2022-11-23T02:43:31.6036939Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_bugs_xqt3kb_v) 2022-11-23T02:43:31.6037461Z 2022-11-23T02:43:31.6037939Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-11-23 02:43:31.602768] 2022-11-23T02:43:31.6040592Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-11-23 02:43:31.603595] 2022-11-23T02:43:34.5313852Z 2022-11-23T02:43:34.5314791Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_verify_skippables 2022-11-23T02:43:34.5317764Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_pzi0cg1d) 2022-11-23T02:43:34.5319106Z ============================= test session starts ============================== 2022-11-23T02:43:34.5320704Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:34.5321489Z cachedir: .pytest_cache 2022-11-23T02:43:34.5322818Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:34.5323782Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:34.5324478Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:34.5325779Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:34.5326661Z collecting ... collected 9 items 2022-11-23T02:43:34.5329761Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated 2022-11-23T02:43:34.5332855Z 2022-11-23T02:43:34.5333358Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%] 2022-11-23T02:43:34.5334392Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%] 2022-11-23T02:43:34.5335444Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%] 2022-11-23T02:43:34.5336498Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%] 2022-11-23T02:43:34.5337523Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%] 2022-11-23T02:43:34.5338777Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%] 2022-11-23T02:43:34.5339958Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%] 2022-11-23T02:43:34.5341069Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%] 2022-11-23T02:43:34.5342188Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%] 2022-11-23T02:43:34.5342804Z 2022-11-23T02:43:34.5343115Z ============================== 9 passed in 0.03s =============================== 2022-11-23T02:43:34.5343524Z 2022-11-23T02:43:34.5344172Z ##[endgroup] 2022-11-23T02:43:34.5345859Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_pzi0cg1d) 2022-11-23T02:43:34.5346820Z 2022-11-23T02:43:34.5347537Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-11-23 02:43:34.531737] 2022-11-23T02:43:34.5348999Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-11-23 02:43:34.532359] 2022-11-23T02:43:37.5818864Z 2022-11-23T02:43:37.5820073Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_stash_pop 2022-11-23T02:43:37.5823934Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_58ehpysd) 2022-11-23T02:43:37.5826083Z ============================= test session starts ============================== 2022-11-23T02:43:37.5828723Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:37.5830216Z cachedir: .pytest_cache 2022-11-23T02:43:37.5833014Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:37.5835058Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:37.5836531Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:37.5839340Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:37.5841320Z collecting ... collected 7 items 2022-11-23T02:43:37.5845514Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none 2022-11-23T02:43:37.5849120Z 2022-11-23T02:43:37.5849949Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED [ 14%] 2022-11-23T02:43:37.5851518Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED [ 28%] 2022-11-23T02:43:37.5853216Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%] 2022-11-23T02:43:37.5855132Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%] 2022-11-23T02:43:37.5856938Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%] 2022-11-23T02:43:37.5858758Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%] 2022-11-23T02:43:37.5860454Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%] 2022-11-23T02:43:37.5861315Z 2022-11-23T02:43:37.5861814Z ============================== 7 passed in 0.03s =============================== 2022-11-23T02:43:37.5862594Z 2022-11-23T02:43:37.5863659Z ##[endgroup] 2022-11-23T02:43:37.5866862Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_58ehpysd) 2022-11-23T02:43:37.5868083Z 2022-11-23T02:43:37.5868897Z Running distributed/pipeline/sync/skip/test_leak ... [2022-11-23 02:43:37.582840] 2022-11-23T02:43:37.5870602Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-11-23 02:43:37.583910] 2022-11-23T02:43:40.9113902Z 2022-11-23T02:43:40.9115125Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_leak 2022-11-23T02:43:40.9118112Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_leak_s20hiiz3) 2022-11-23T02:43:40.9119666Z ============================= test session starts ============================== 2022-11-23T02:43:40.9121705Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:40.9122659Z cachedir: .pytest_cache 2022-11-23T02:43:40.9124274Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:40.9125448Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:40.9126302Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:40.9127909Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:40.9128981Z collecting ... collected 8 items 2022-11-23T02:43:40.9134210Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] 2022-11-23T02:43:40.9137200Z 2022-11-23T02:43:40.9138212Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%] 2022-11-23T02:43:40.9139882Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%] 2022-11-23T02:43:40.9141582Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%] 2022-11-23T02:43:40.9143921Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%] 2022-11-23T02:43:40.9145645Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%] 2022-11-23T02:43:40.9147301Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%] 2022-11-23T02:43:40.9148585Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%] 2022-11-23T02:43:40.9149871Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%] 2022-11-23T02:43:40.9150563Z 2022-11-23T02:43:40.9150962Z ============================== 8 passed in 0.27s =============================== 2022-11-23T02:43:40.9151458Z 2022-11-23T02:43:40.9152285Z ##[endgroup] 2022-11-23T02:43:40.9154147Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_leak_s20hiiz3) 2022-11-23T02:43:40.9155213Z 2022-11-23T02:43:40.9156031Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-11-23 02:43:40.911956] 2022-11-23T02:43:40.9157798Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-11-23 02:43:40.912756] 2022-11-23T02:43:50.6956130Z 2022-11-23T02:43:50.6958050Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_gpipe 2022-11-23T02:43:50.6961055Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_5ncqefa4) 2022-11-23T02:43:50.6962536Z ============================= test session starts ============================== 2022-11-23T02:43:50.6964187Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T02:43:50.6965183Z cachedir: .pytest_cache 2022-11-23T02:43:50.6966773Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T02:43:50.6967941Z torch: 1.14.0a0+git1cfd385 2022-11-23T02:43:50.6968777Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T02:43:50.6970341Z plugins: xdist-3.0.2, rerunfailures-10.3, hypothesis-5.35.1, shard-0.1.2, flakefinder-1.1.0, xdoctest-1.0.2 2022-11-23T02:43:50.6971416Z collecting ... collected 13 items 2022-11-23T02:43:50.6977942Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip 2022-11-23T02:43:50.6981578Z 2022-11-23T02:43:50.6982394Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] PASSED [ 7%] 2022-11-23T02:43:50.6983848Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] PASSED [ 15%] 2022-11-23T02:43:50.6985268Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] PASSED [ 23%] 2022-11-23T02:43:50.6986730Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%] 2022-11-23T02:43:50.6988179Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] PASSED [ 38%] 2022-11-23T02:43:50.6989886Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] PASSED [ 46%] 2022-11-23T02:43:50.6991325Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] PASSED [ 53%] 2022-11-23T02:43:50.6992765Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%] 2022-11-23T02:43:50.6994255Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] PASSED [ 69%] 2022-11-23T02:43:50.6995749Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] PASSED [ 76%] 2022-11-23T02:43:50.6997232Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] PASSED [ 84%] 2022-11-23T02:43:50.6998747Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%] 2022-11-23T02:43:50.6999932Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED [100%] 2022-11-23T02:43:50.7000826Z 2022-11-23T02:43:50.7001248Z ======================== 10 passed, 3 skipped in 6.04s ========================= 2022-11-23T02:43:50.7001779Z 2022-11-23T02:43:50.7002542Z ##[endgroup] 2022-11-23T02:43:50.7004397Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_5ncqefa4) 2022-11-23T02:43:50.7005464Z 2022-11-23T02:43:50.7006485Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-11-23 02:43:50.696125] 2022-11-23T02:43:50.7008761Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:43:50.696783] 2022-11-23T02:46:46.2832833Z 2022-11-23T02:46:46.2834175Z Expand the folded group to see the log file of distributed/optim/test_zero_redundancy_optimizer 2022-11-23T02:46:46.2836838Z ##[group]PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/pytorch/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_5driyvb7) 2022-11-23T02:46:46.2839367Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer 2022-11-23T02:46:46.2840472Z 2022-11-23T02:46:46.2840767Z Running tests... 2022-11-23T02:46:46.2841978Z ---------------------------------------------------------------------- 2022-11-23T02:46:46.2843263Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.2847996Z Check that ZeroRedundancyOptimizer properly handles adding a new ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35622 2022-11-23T02:46:46.2849916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35623 2022-11-23T02:46:46.2852755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2854429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2856123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2857914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2859422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.2860771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.2862540Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2863731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2865376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2866741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2867897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.2869741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.2871785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2873973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2874986Z ok (3.839s) 2022-11-23T02:46:46.2875939Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.2878102Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 50 2022-11-23T02:46:46.2879511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35742 2022-11-23T02:46:46.2881121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35743 2022-11-23T02:46:46.2882838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2884050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2885649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2887152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2888330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.2889803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.2891656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2892842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2894440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2895702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2896871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.2899307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.2901132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2903035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2904055Z ok (3.234s) 2022-11-23T02:46:46.2905003Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.2906827Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 49 2022-11-23T02:46:46.2908202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35862 2022-11-23T02:46:46.2909375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35863 2022-11-23T02:46:46.2911049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2912249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2913835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2915072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2916233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.2917537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.2919263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2920785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2922386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2923642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2924799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.2926092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.2927926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2958807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2959910Z ok (3.134s) 2022-11-23T02:46:46.2961072Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.2962979Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 48 2022-11-23T02:46:46.2964377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35982 2022-11-23T02:46:46.2965478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35983 2022-11-23T02:46:46.2966153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2966619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2967250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2967747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2968210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.2968735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.2969427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2969895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2970507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2970999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2971460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.2971975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.2972692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2973451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2973856Z ok (3.333s) 2022-11-23T02:46:46.2974233Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.2974938Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 47 2022-11-23T02:46:46.2975479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36102 2022-11-23T02:46:46.2975952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36103 2022-11-23T02:46:46.2976609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2977083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2977700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2978261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2978702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.2979226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.2979916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2980387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2981013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2981507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2981964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.2982468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.2983181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2983967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2984377Z ok (3.133s) 2022-11-23T02:46:46.2984752Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.2985467Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 46 2022-11-23T02:46:46.2986010Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36222 2022-11-23T02:46:46.2986484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36223 2022-11-23T02:46:46.2987125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2987594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2988219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2988711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2989169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.2989683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.2990362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2990812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2991431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2991920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2992374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.2992887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.2993611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2994359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.2994750Z ok (3.134s) 2022-11-23T02:46:46.2995122Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.2995832Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 45 2022-11-23T02:46:46.2996373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36342 2022-11-23T02:46:46.2996899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36343 2022-11-23T02:46:46.2997550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.2998023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.2998634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.2999126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.2999583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3000093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3000861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3001337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3001959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3002455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3002960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3003476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3004186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3004938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3005348Z ok (3.233s) 2022-11-23T02:46:46.3005723Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3006441Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 44 2022-11-23T02:46:46.3006973Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36462 2022-11-23T02:46:46.3007447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36463 2022-11-23T02:46:46.3008098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3008565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3009189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3009681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3010137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3010633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3011311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3011778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3012406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3012895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3013351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3013862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3014574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3015362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3015767Z ok (3.139s) 2022-11-23T02:46:46.3016143Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3016862Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 43 2022-11-23T02:46:46.3017401Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36582 2022-11-23T02:46:46.3017872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36583 2022-11-23T02:46:46.3018522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3018977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3019596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3020091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3020548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3021062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3021787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3022251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3022862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3023353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3023804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3024313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3025030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3025779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3026184Z ok (3.334s) 2022-11-23T02:46:46.3026541Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3027253Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 42 2022-11-23T02:46:46.3027790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36702 2022-11-23T02:46:46.3028260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36703 2022-11-23T02:46:46.3028913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3029385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3030008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3030495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3030941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3031453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3032135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3032603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3033223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3033707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3034204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3034700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3035418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3036164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3036570Z ok (3.334s) 2022-11-23T02:46:46.3036943Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3037651Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 41 2022-11-23T02:46:46.3038189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36822 2022-11-23T02:46:46.3038647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36823 2022-11-23T02:46:46.3039300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3039770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3040613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3041163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3041623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3042140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3042829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3043281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3043906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3044399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3044864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3045382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3046096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3046842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3047233Z ok (3.234s) 2022-11-23T02:46:46.3047605Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3048320Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 40 2022-11-23T02:46:46.3048864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36942 2022-11-23T02:46:46.3049339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36943 2022-11-23T02:46:46.3049996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3050466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3051074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3051563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3052019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3052533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3053305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3053769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3054398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3054889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3055333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3055850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3056565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3057306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3057712Z ok (3.244s) 2022-11-23T02:46:46.3058085Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3058852Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 39 2022-11-23T02:46:46.3059386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37062 2022-11-23T02:46:46.3059853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37063 2022-11-23T02:46:46.3060515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3060982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3061605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3062097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3062552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3063051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3063733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3064199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3064820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3065311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3065769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3066280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3066986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3067731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3068141Z ok (3.332s) 2022-11-23T02:46:46.3068516Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3069227Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 38 2022-11-23T02:46:46.3069765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37182 2022-11-23T02:46:46.3070236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37183 2022-11-23T02:46:46.3070883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3071384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3072009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3072501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3072959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3073472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3074152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3074618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3075224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3075708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3076168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3076678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3077437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3078194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3078600Z ok (3.033s) 2022-11-23T02:46:46.3078962Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3079671Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 37 2022-11-23T02:46:46.3080299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37302 2022-11-23T02:46:46.3080782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37303 2022-11-23T02:46:46.3081439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3081907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3082529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3083006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3083461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3083981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3084658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3085127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3085755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3086249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3086706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3087206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3087919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3088661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3089062Z ok (3.032s) 2022-11-23T02:46:46.3089440Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3090151Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 36 2022-11-23T02:46:46.3090761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37422 2022-11-23T02:46:46.3091214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37423 2022-11-23T02:46:46.3091871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3092343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3092969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3093458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3093920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3094432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3095103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3095568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3096237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3096731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3097187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3097699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3098415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3099157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3099548Z ok (3.233s) 2022-11-23T02:46:46.3099925Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3100638Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 35 2022-11-23T02:46:46.3101177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37542 2022-11-23T02:46:46.3101647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37543 2022-11-23T02:46:46.3102300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3102771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3103382Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3103875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3104335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3104855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3105536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3106003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3106612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3107103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3107559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3108077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3108839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3109590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3109993Z ok (3.132s) 2022-11-23T02:46:46.3110369Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3111066Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 34 2022-11-23T02:46:46.3111604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37662 2022-11-23T02:46:46.3112071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37663 2022-11-23T02:46:46.3112721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3113185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3113810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3114298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3114783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3115297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3115974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3116442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3117065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3117554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3118017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3118518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3119235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3119976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3120450Z ok (3.232s) 2022-11-23T02:46:46.3120830Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3121549Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 33 2022-11-23T02:46:46.3122088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37782 2022-11-23T02:46:46.3122544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37783 2022-11-23T02:46:46.3123208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3123679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3124306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3124799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3125273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3125790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3126474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3126928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3127615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3128101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3128560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3129079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3129800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3130549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3130940Z ok (3.132s) 2022-11-23T02:46:46.3131315Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3132027Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 32 2022-11-23T02:46:46.3132571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37902 2022-11-23T02:46:46.3133043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37903 2022-11-23T02:46:46.3133745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3134216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3134832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3135326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3135788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3136299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3136986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3137455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3138079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3138567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3139011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3139519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3140233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3140985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3141395Z ok (3.233s) 2022-11-23T02:46:46.3141772Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3142483Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 31 2022-11-23T02:46:46.3143015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38022 2022-11-23T02:46:46.3143483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38023 2022-11-23T02:46:46.3144137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3144606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3145235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3145723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3146228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3146724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3147411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3147874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3148498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3148986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3149442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3149952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3150658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3151398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3151804Z ok (3.333s) 2022-11-23T02:46:46.3152223Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3152935Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 30 2022-11-23T02:46:46.3153475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38142 2022-11-23T02:46:46.3153942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38143 2022-11-23T02:46:46.3154598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3155052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3155677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3156161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3156610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3157122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3157797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3158258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3158863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3159349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3159804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3160392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3161116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3161869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3162276Z ok (3.434s) 2022-11-23T02:46:46.3162640Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3163354Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 29 2022-11-23T02:46:46.3163897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38262 2022-11-23T02:46:46.3164371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38263 2022-11-23T02:46:46.3165153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3165627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3166257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3166750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3167195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3167709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3168392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3168859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3169481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3169973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3170428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3170981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3171696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3172440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3172849Z ok (3.134s) 2022-11-23T02:46:46.3173223Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3173929Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 28 2022-11-23T02:46:46.3174462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38382 2022-11-23T02:46:46.3174916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38383 2022-11-23T02:46:46.3175571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3176038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3176657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3177144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3177594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3178107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3178790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3179244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3179868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3180359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3180817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3181328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3182041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3182792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3183230Z ok (3.133s) 2022-11-23T02:46:46.3183605Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3184315Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 27 2022-11-23T02:46:46.3184856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38502 2022-11-23T02:46:46.3185326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38503 2022-11-23T02:46:46.3185980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3186447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3187051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3187539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3187997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3188510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3189192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3189719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3190346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3190822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3191278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3191784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3192496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3193246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3193655Z ok (3.233s) 2022-11-23T02:46:46.3194031Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3194746Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 26 2022-11-23T02:46:46.3195276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38622 2022-11-23T02:46:46.3195751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38623 2022-11-23T02:46:46.3196405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3196873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3197501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3197997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3198460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3198962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3199642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3200109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3200818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3201307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3201768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3202352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3203057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3203810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3204220Z ok (3.033s) 2022-11-23T02:46:46.3204595Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3205310Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 25 2022-11-23T02:46:46.3205855Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38742 2022-11-23T02:46:46.3206334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38743 2022-11-23T02:46:46.3206982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3207453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3208071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3208611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3209071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3209584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3210276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3210750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3211356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3211850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3212304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3212826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3213542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3214295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3214701Z ok (3.433s) 2022-11-23T02:46:46.3215063Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3215770Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 24 2022-11-23T02:46:46.3216314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38862 2022-11-23T02:46:46.3216781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38863 2022-11-23T02:46:46.3217432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3217905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3218526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3219002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3219461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3219980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3220655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3221170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3221793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3222283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3222741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3223239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3223949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3224695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3225099Z ok (3.133s) 2022-11-23T02:46:46.3225475Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3226187Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 23 2022-11-23T02:46:46.3226730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38982 2022-11-23T02:46:46.3227238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38983 2022-11-23T02:46:46.3227889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3228354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3228978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3229465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3229923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3230442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3231107Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3231573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3232197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3232687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3233146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3233654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3234365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3235109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3235501Z ok (3.133s) 2022-11-23T02:46:46.3235874Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3236585Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 22 2022-11-23T02:46:46.3237124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39102 2022-11-23T02:46:46.3237593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39103 2022-11-23T02:46:46.3238245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3238713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3239324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3239865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3240413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3240937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3241630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3242099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3242721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3243196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3243652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3244171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3244885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3245697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3246106Z ok (3.028s) 2022-11-23T02:46:46.3246478Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3247178Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 21 2022-11-23T02:46:46.3247717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39222 2022-11-23T02:46:46.3248186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39223 2022-11-23T02:46:46.3248837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3249305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3249924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3250410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3250869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3251365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3252039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3252505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3253124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3253612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3254072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3254592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3255292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3256036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3256436Z ok (3.133s) 2022-11-23T02:46:46.3256808Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3257516Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 20 2022-11-23T02:46:46.3258059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39342 2022-11-23T02:46:46.3258586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39343 2022-11-23T02:46:46.3259231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3259703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3260323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3260813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3261269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3261781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3262460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3262934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3263544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3264083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3264542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3265058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3265773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3266515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3266915Z ok (3.233s) 2022-11-23T02:46:46.3267274Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3267995Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 19 2022-11-23T02:46:46.3268529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39462 2022-11-23T02:46:46.3269003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39463 2022-11-23T02:46:46.3269659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3270139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3270769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3271250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3271715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3272234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3272930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3273406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3274049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3274537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3274977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3275489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3276203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3277001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3277406Z ok (3.033s) 2022-11-23T02:46:46.3277783Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3278498Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 18 2022-11-23T02:46:46.3279040Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39582 2022-11-23T02:46:46.3279499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39583 2022-11-23T02:46:46.3280196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3280668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3281293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3281781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3282235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3282807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3283484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3283945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3284556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3285043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3285495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3286008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3286719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3287466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3287860Z ok (3.232s) 2022-11-23T02:46:46.3288232Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3288939Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 17 2022-11-23T02:46:46.3289475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39702 2022-11-23T02:46:46.3289941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39703 2022-11-23T02:46:46.3290597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3291064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3291672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3292154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3292605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3293113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3293791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3294251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3294868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3295403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3296007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3296507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3297215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3297963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3298364Z ok (3.133s) 2022-11-23T02:46:46.3298733Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3299426Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 16 2022-11-23T02:46:46.3299961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39822 2022-11-23T02:46:46.3300425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39823 2022-11-23T02:46:46.3301069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3301587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3302206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3302691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3303130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3303637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3304314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3304776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3305394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3305875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3306329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3306827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3307523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3308267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3308671Z ok (3.233s) 2022-11-23T02:46:46.3309041Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3309751Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 15 2022-11-23T02:46:46.3310288Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39942 2022-11-23T02:46:46.3310758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39943 2022-11-23T02:46:46.3311399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3311867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3312483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3312972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3313427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3313992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3314669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3315117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3315732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3316219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3316676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3317185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3317898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3318641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3319049Z ok (3.233s) 2022-11-23T02:46:46.3319409Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3320242Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 14 2022-11-23T02:46:46.3320805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40062 2022-11-23T02:46:46.3321279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40063 2022-11-23T02:46:46.3321942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3322416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3323041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3323524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3324099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3324808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3325546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3326200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3326921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3340312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3340802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3341309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3342112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3342859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3343270Z ok (3.532s) 2022-11-23T02:46:46.3343649Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3344363Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 13 2022-11-23T02:46:46.3344903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40182 2022-11-23T02:46:46.3345367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40183 2022-11-23T02:46:46.3346026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3346678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3347307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3347796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3348258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3348776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3349456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3349909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3350528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3351020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3351479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3351992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3352780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3353525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3353916Z ok (3.132s) 2022-11-23T02:46:46.3354287Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3354995Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 12 2022-11-23T02:46:46.3355536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40302 2022-11-23T02:46:46.3356009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40303 2022-11-23T02:46:46.3356662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3357130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3357742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3358236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3358694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3359205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3359885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3360457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3361092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3361582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3362027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3362536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3363255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3364002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3364404Z ok (3.232s) 2022-11-23T02:46:46.3364779Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3365559Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 11 2022-11-23T02:46:46.3366091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40422 2022-11-23T02:46:46.3366559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40423 2022-11-23T02:46:46.3367213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3367682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3368300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3368788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3369243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3369747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3370423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3370882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3371549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3372043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3372500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3373011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3373716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3374455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3374862Z ok (3.633s) 2022-11-23T02:46:46.3375234Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3375950Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 10 2022-11-23T02:46:46.3376492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40542 2022-11-23T02:46:46.3376963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40543 2022-11-23T02:46:46.3377613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3378063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3378684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3379172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3379624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3380133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3380814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3381280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3381883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3382369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3382823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3383335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3384104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3384859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3385261Z ok (3.835s) 2022-11-23T02:46:46.3385629Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3386333Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 9 2022-11-23T02:46:46.3386866Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40662 2022-11-23T02:46:46.3387335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40663 2022-11-23T02:46:46.3387985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3388454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3389071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3389550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3390055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3390566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3391247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3391706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3392319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3392801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3393255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3393748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3394458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3395354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3395759Z ok (3.434s) 2022-11-23T02:46:46.3396133Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3396838Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 8 2022-11-23T02:46:46.3397372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40782 2022-11-23T02:46:46.3397829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40783 2022-11-23T02:46:46.3398480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3398942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3399560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3400048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3400602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3401113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3401776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3402236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3402921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3403410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3403866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3404373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3405092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3405835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3406228Z ok (3.637s) 2022-11-23T02:46:46.3406599Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3407301Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 7 2022-11-23T02:46:46.3407839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40902 2022-11-23T02:46:46.3408311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40903 2022-11-23T02:46:46.3409020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3409489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3410093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3410573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3411032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3411541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3412220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3412686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3413310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3413784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3414241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3414748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3415462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3416200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3416605Z ok (3.333s) 2022-11-23T02:46:46.3416977Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3417669Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 6 2022-11-23T02:46:46.3418208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41022 2022-11-23T02:46:46.3418676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41023 2022-11-23T02:46:46.3419323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3419785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3420405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3420891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3421390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3421881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3422555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3423017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3423638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3424125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3424576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3425086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3425788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3426522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3426923Z ok (3.234s) 2022-11-23T02:46:46.3427336Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3428049Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 5 2022-11-23T02:46:46.3428585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41142 2022-11-23T02:46:46.3429053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41143 2022-11-23T02:46:46.3429686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3430149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3430769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3431247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3431707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3432220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3432902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3433363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3433971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3434453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3434905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3435415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3436126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3436872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3437277Z ok (3.033s) 2022-11-23T02:46:46.3437635Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3438338Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 4 2022-11-23T02:46:46.3438870Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41262 2022-11-23T02:46:46.3439339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41263 2022-11-23T02:46:46.3440042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3440578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3441210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3441686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3442142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3442650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3443330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3443792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3444412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3444896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3445336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3445896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3446617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3447356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3447764Z ok (3.233s) 2022-11-23T02:46:46.3448137Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3448840Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 3 2022-11-23T02:46:46.3449376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41382 2022-11-23T02:46:46.3449830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41383 2022-11-23T02:46:46.3450477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3450945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3451562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3452046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3452503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3453018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3453687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3454142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3454756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3455246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3455697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3456205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3456912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3457644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3458117Z ok (3.335s) 2022-11-23T02:46:46.3458483Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3459188Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 2 2022-11-23T02:46:46.3459726Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41502 2022-11-23T02:46:46.3460193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41503 2022-11-23T02:46:46.3460841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3461305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3461912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3462397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3462852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3463362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3464085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3464546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3465171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3465647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3466098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3466605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3467312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3468056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3468458Z ok (3.033s) 2022-11-23T02:46:46.3468828Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3469516Z Check that ZeroRedundancyOptimizer properly handles adding a new ... test_add_param_group succeeded - num_retries_left: 1 2022-11-23T02:46:46.3470047Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41622 2022-11-23T02:46:46.3470508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41623 2022-11-23T02:46:46.3471157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3471622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3472249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3472733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3473172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3473682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3474354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3474819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3475438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3475925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3476379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3476933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3477636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3478379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3478777Z ok (3.334s) 2022-11-23T02:46:46.3479152Z test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3479842Z Check the state consolidation mechanism and the state dict exposed ... test_add_param_group succeeded - num_retries_left: 0 2022-11-23T02:46:46.3480627Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.006s) 2022-11-23T02:46:46.3481330Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3482339Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:46:46.3483195Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3484171Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3484958Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3485949Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:46:46.3486722Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3487694Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3488471Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3489444Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3490228Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3491198Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3491973Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3492938Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3493704Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3494860Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3495633Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3496608Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:46:46.3497385Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3498359Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3499134Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3500136Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3500905Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3501875Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3502653Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3503630Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3504406Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3505368Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3506136Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3507105Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3507873Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3508831Z Check that overlapping DDP with ZeRO using the given method determined ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3509529Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3510429Z When combined with DDP, check that a local optimizer gives the same ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:46:46.3511175Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3512064Z When combined with DDP, check that a local optimizer gives the same ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:46:46.3512760Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3513659Z When combined with DDP, check that a local optimizer gives the same ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:46:46.3514341Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3515230Z When combined with DDP, check that a local optimizer gives the same ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:46:46.3515924Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3516855Z When combined with DDP, check that a local optimizer gives the same ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:46:46.3517535Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3518418Z When combined with DDP, check that a local optimizer gives the same ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T02:46:46.3519017Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3519819Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3520511Z test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3521382Z Check parity between constructing ZeRO with multiple parameter groups ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:46:46.3522030Z test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3522610Z Check that ZeroRedundancyOptimizer works with a non-default process ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:46:46.3522831Z test_sharding (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3523350Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41742 2022-11-23T02:46:46.3523574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41743 2022-11-23T02:46:46.3523982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3524159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3524576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3524770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3525005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3525258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:46:46.3525660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3525838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3526315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3526507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3526741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3526982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:46:46.3527415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3527851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:46:46.3527948Z ok (3.026s) 2022-11-23T02:46:46.3528167Z test_step (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3528747Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.006s) 2022-11-23T02:46:46.3528992Z test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3529630Z Check that ZeroRedundancyOptimizer properly exposes the ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.005s) 2022-11-23T02:46:46.3529861Z test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3530416Z Check that the ZeRO join hook allows training with uneven inputs ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:46:46.3530644Z test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3531193Z Check that the ZeRO join hook allows training with uneven inputs ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T02:46:46.3531491Z test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3532043Z Check that ZeRO works with model parallelism where the model's ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3532332Z test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T02:46:46.3532811Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41862 2022-11-23T02:46:46.3533031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41863 2022-11-23T02:46:46.3533436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3533609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3534026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3534217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3534451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:46:46.3534846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:46:46.3535023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:46:46.3535428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:46:46.3535618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:46:46.3535834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:46:46.3536026Z skip: Need at least 4 CUDA devices (3.224s) 2022-11-23T02:46:46.3536253Z test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T02:46:46.3536887Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.006s) 2022-11-23T02:46:46.3537118Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T02:46:46.3537662Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:46:46.3537904Z test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T02:46:46.3538483Z Check that ZeroRedundancyOptimizer raises an exception if the input ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3538708Z test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T02:46:46.3539282Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:46:46.3539568Z test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T02:46:46.3540149Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3540381Z test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T02:46:46.3540937Z Check that the ``step(**kwargs)`` interface is properly exposed. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3541177Z test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T02:46:46.3541727Z Check that the ``step()`` method (without closure) is handled as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3541945Z test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T02:46:46.3542477Z Check that the ``zero_grad`` method is properly handled. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:46.3542487Z 2022-11-23T02:46:46.3542780Z ---------------------------------------------------------------------- 2022-11-23T02:46:46.3542892Z Ran 92 tests in 172.270s 2022-11-23T02:46:46.3542899Z 2022-11-23T02:46:46.3542997Z OK (skipped=40) 2022-11-23T02:46:46.3543004Z 2022-11-23T02:46:46.3543118Z Generating XML reports... 2022-11-23T02:46:46.3543743Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221123024352.xml 2022-11-23T02:46:46.3544350Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221123024352.xml 2022-11-23T02:46:46.3544357Z 2022-11-23T02:46:46.3544964Z ##[endgroup] 2022-11-23T02:46:46.3545621Z FINISHED PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/pytorch/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_5driyvb7) 2022-11-23T02:46:46.3545629Z 2022-11-23T02:46:46.3545924Z Running distributed/nn/jit/test_instantiator ... [2022-11-23 02:46:46.287485] 2022-11-23T02:46:46.3546533Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:46:46.288700] 2022-11-23T02:46:49.7734972Z 2022-11-23T02:46:49.7736344Z Expand the folded group to see the log file of distributed/nn/jit/test_instantiator 2022-11-23T02:46:49.7739244Z ##[group]PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/pytorch/test/test-reports/distributed-nn-jit-test_instantiator_h9up0kv0) 2022-11-23T02:46:49.7741504Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator 2022-11-23T02:46:49.7742272Z 2022-11-23T02:46:49.7742535Z Running tests... 2022-11-23T02:46:49.7743659Z ---------------------------------------------------------------------- 2022-11-23T02:46:49.7745763Z test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.484s) 2022-11-23T02:46:49.7748305Z test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:49.7750961Z test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:49.7752007Z 2022-11-23T02:46:49.7752737Z ---------------------------------------------------------------------- 2022-11-23T02:46:49.7753593Z Ran 3 tests in 0.487s 2022-11-23T02:46:49.7753990Z 2022-11-23T02:46:49.7754456Z OK (skipped=3) 2022-11-23T02:46:49.7754840Z 2022-11-23T02:46:49.7755109Z Generating XML reports... 2022-11-23T02:46:49.7756790Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221123024647.xml 2022-11-23T02:46:49.7757731Z 2022-11-23T02:46:49.7758481Z ##[endgroup] 2022-11-23T02:46:49.7760356Z FINISHED PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/pytorch/test/test-reports/distributed-nn-jit-test_instantiator_h9up0kv0) 2022-11-23T02:46:49.7761320Z 2022-11-23T02:46:49.7762043Z Running distributed/fsdp/test_utils ... [2022-11-23 02:46:49.774082] 2022-11-23T02:46:49.7764087Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:46:49.774837] 2022-11-23T02:46:53.3098387Z 2022-11-23T02:46:53.3099321Z Expand the folded group to see the log file of distributed/fsdp/test_utils 2022-11-23T02:46:53.3101783Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_utils_z8u6rt7a) 2022-11-23T02:46:53.3104105Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils 2022-11-23T02:46:53.3104831Z 2022-11-23T02:46:53.3105063Z Running tests... 2022-11-23T02:46:53.3106196Z ---------------------------------------------------------------------- 2022-11-23T02:46:53.3107298Z test_module_wrap_policy (__main__.TestGetSubmoduleToStates) 2022-11-23T02:46:53.3109346Z Tests the module wrap policy on a nested model with buffers and a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.544s) 2022-11-23T02:46:53.3111696Z test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:46:53.3114005Z test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:46:53.3116317Z test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:46:53.3117623Z test_packed_sequence (__main__.TestUtils) 2022-11-23T02:46:53.3119464Z Test to ensure RNN packed sequences are modified correctly. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:53.3122545Z test_replace_by_prefix (__main__.TestUtils) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:53.3123455Z 2022-11-23T02:46:53.3124201Z ---------------------------------------------------------------------- 2022-11-23T02:46:53.3125060Z Ran 6 tests in 0.554s 2022-11-23T02:46:53.3125477Z 2022-11-23T02:46:53.3125723Z OK (skipped=6) 2022-11-23T02:46:53.3126103Z 2022-11-23T02:46:53.3126392Z Generating XML reports... 2022-11-23T02:46:53.3128095Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20221123024651.xml 2022-11-23T02:46:53.3130153Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221123024651.xml 2022-11-23T02:46:53.3130996Z 2022-11-23T02:46:53.3131759Z ##[endgroup] 2022-11-23T02:46:53.3133324Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_utils_z8u6rt7a) 2022-11-23T02:46:53.3134213Z 2022-11-23T02:46:53.3135030Z Running distributed/fsdp/test_fsdp_use_orig_params ... [2022-11-23 02:46:53.310504] 2022-11-23T02:46:53.3137416Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:46:53.311379] 2022-11-23T02:46:56.9075377Z 2022-11-23T02:46:56.9076779Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_use_orig_params 2022-11-23T02:46:56.9081035Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_10jpzw5w) 2022-11-23T02:46:56.9083540Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params 2022-11-23T02:46:56.9084408Z 2022-11-23T02:46:56.9084661Z Running tests... 2022-11-23T02:46:56.9085799Z ---------------------------------------------------------------------- 2022-11-23T02:46:56.9086977Z test_named_parameters_in_forward (__main__.TestFSDPUseOrigParamsFQNs) 2022-11-23T02:46:56.9090306Z Tests that calling ``named_parameters()`` during forward returns FQNs ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.525s) 2022-11-23T02:46:56.9091825Z test_param_and_buffer_names (__main__.TestFSDPUseOrigParamsFQNs) 2022-11-23T02:46:56.9093884Z Tests that, for ``use_orig_params=True``, the parameter and buffer ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9095739Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T02:46:56.9099229Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9101064Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T02:46:56.9103400Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9105245Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T02:46:56.9107598Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9109339Z test_diff_hyperparams_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T02:46:56.9111608Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9113742Z test_diff_hyperparams_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T02:46:56.9116019Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9117752Z test_diff_hyperparams_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T02:46:56.9120045Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9121809Z test_diff_trainability (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T02:46:56.9123998Z Tests FSDP parity with DDP when using multiple parameter groups and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9125646Z test_multiple_optimizers (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T02:46:56.9127982Z Tests using two optimizers where only one sets gradients to ``None``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9129574Z test_access_params_after_forward (__main__.TestFSDPUseOrigParamsParamAccess) 2022-11-23T02:46:56.9131723Z Tests that accessing the original parameters after the forward but ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9133380Z test_multiple_forward_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T02:46:56.9135539Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9137183Z test_multiple_forward_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T02:46:56.9139301Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9140962Z test_summon_between_two_forwards_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T02:46:56.9143121Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9144757Z test_summon_between_two_forwards_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T02:46:56.9146907Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9148366Z test_grad_writeback (__main__.TestFSDPUseOrigParamsWriteback) 2022-11-23T02:46:56.9150447Z Tests that changes to the original parameters' gradients are written ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9151979Z test_param_writeback (__main__.TestFSDPUseOrigParamsWriteback) 2022-11-23T02:46:56.9154049Z Tests that changes to the original parameters are written back. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:46:56.9156593Z test_writeback_shape_mismatch (__main__.TestFSDPUseOrigParamsWriteback) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:46:56.9157698Z 2022-11-23T02:46:56.9158418Z ---------------------------------------------------------------------- 2022-11-23T02:46:56.9159420Z Ran 18 tests in 0.543s 2022-11-23T02:46:56.9159817Z 2022-11-23T02:46:56.9160066Z OK (skipped=18) 2022-11-23T02:46:56.9160559Z 2022-11-23T02:46:56.9160852Z Generating XML reports... 2022-11-23T02:46:56.9162701Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123024654.xml 2022-11-23T02:46:56.9165326Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123024654.xml 2022-11-23T02:46:56.9167951Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123024654.xml 2022-11-23T02:46:56.9170548Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123024654.xml 2022-11-23T02:46:56.9173053Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123024654.xml 2022-11-23T02:46:56.9174163Z 2022-11-23T02:46:56.9174974Z ##[endgroup] 2022-11-23T02:46:56.9176727Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_10jpzw5w) 2022-11-23T02:46:56.9177869Z 2022-11-23T02:46:56.9178668Z Running distributed/fsdp/test_fsdp_traversal ... [2022-11-23 02:46:56.908415] 2022-11-23T02:46:56.9180804Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:46:56.909293] 2022-11-23T02:47:00.4482976Z 2022-11-23T02:47:00.4484340Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal 2022-11-23T02:47:00.4487477Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_traversal_muty7j9k) 2022-11-23T02:47:00.4489789Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal 2022-11-23T02:47:00.4490545Z 2022-11-23T02:47:00.4490799Z Running tests... 2022-11-23T02:47:00.4492057Z ---------------------------------------------------------------------- 2022-11-23T02:47:00.4494030Z test_fsdp_modules (__main__.TestTraversal) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.488s) 2022-11-23T02:47:00.4494964Z 2022-11-23T02:47:00.4495684Z ---------------------------------------------------------------------- 2022-11-23T02:47:00.4496534Z Ran 1 test in 0.488s 2022-11-23T02:47:00.4496930Z 2022-11-23T02:47:00.4497172Z OK (skipped=1) 2022-11-23T02:47:00.4497548Z 2022-11-23T02:47:00.4497837Z Generating XML reports... 2022-11-23T02:47:00.4499500Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123024658.xml 2022-11-23T02:47:00.4500427Z 2022-11-23T02:47:00.4501157Z ##[endgroup] 2022-11-23T02:47:00.4502843Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_traversal_muty7j9k) 2022-11-23T02:47:00.4503796Z 2022-11-23T02:47:00.4504621Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-11-23 02:47:00.449040] 2022-11-23T02:47:00.4506832Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:47:00.449926] 2022-11-23T02:47:04.0802027Z 2022-11-23T02:47:04.0803193Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_summon_full_params 2022-11-23T02:47:04.0806523Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_6f1srp5y) 2022-11-23T02:47:04.0809395Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params 2022-11-23T02:47:04.0810015Z 2022-11-23T02:47:04.0810213Z Running tests... 2022-11-23T02:47:04.0811085Z ---------------------------------------------------------------------- 2022-11-23T02:47:04.0812964Z test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.501s) 2022-11-23T02:47:04.0815159Z test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0817446Z test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams) 2022-11-23T02:47:04.0819194Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0820480Z test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams) 2022-11-23T02:47:04.0822185Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0823849Z test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams) 2022-11-23T02:47:04.0826424Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0827720Z test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams) 2022-11-23T02:47:04.0829446Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0831739Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.013s) 2022-11-23T02:47:04.0834261Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.006s) 2022-11-23T02:47:04.0836727Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0839153Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0841714Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0844164Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0846606Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0849023Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0851623Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0854047Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0856450Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0858846Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0861430Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0863862Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0866266Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0868665Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0870051Z test_raises_rank0_with_writeback (__main__.TestSummonFullParams) 2022-11-23T02:47:04.0871693Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0874072Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0876666Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0879264Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0881973Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0884572Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0887136Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0889826Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0892401Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:04.0894585Z test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0896872Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0899402Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0901854Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0904281Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0906703Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0909114Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0911513Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0913940Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0916246Z test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0918454Z test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0920627Z test_summon_full_param_writeback (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0922990Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0924141Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0925223Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0926291Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:04.0927339Z test_summon_full_params_respects_reshard_after_forward (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0928297Z test_summon_single_param (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0928847Z test_with_grads_core (__main__.TestSummonFullParams) 2022-11-23T02:47:04.0929640Z Tests the core usage of ``summon_full_params(with_grads=True)``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0930202Z test_with_grads_none_grads (__main__.TestSummonFullParams) 2022-11-23T02:47:04.0930980Z Tests that if all ranks' ``FlatParameter`` has ``None`` gradient, then ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0931953Z test_summon_full_param_writeback (__main__.TestSummonFullParamsNoShard) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:04.0932374Z 2022-11-23T02:47:04.0932666Z ---------------------------------------------------------------------- 2022-11-23T02:47:04.0933011Z Ran 52 tests in 0.611s 2022-11-23T02:47:04.0933172Z 2022-11-23T02:47:04.0933274Z OK (skipped=52) 2022-11-23T02:47:04.0933427Z 2022-11-23T02:47:04.0933613Z Generating XML reports... 2022-11-23T02:47:04.0934338Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221123024702.xml 2022-11-23T02:47:04.0935365Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221123024702.xml 2022-11-23T02:47:04.0935801Z 2022-11-23T02:47:04.0954835Z ##[endgroup] 2022-11-23T02:47:04.0955719Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_6f1srp5y) 2022-11-23T02:47:04.0956133Z 2022-11-23T02:47:04.0956492Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-11-23 02:47:04.081205] 2022-11-23T02:47:04.0957346Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:47:04.082093] 2022-11-23T02:47:07.6206717Z 2022-11-23T02:47:07.6208135Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-11-23T02:47:07.6211662Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_lrjvycmd) 2022-11-23T02:47:07.6214077Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler 2022-11-23T02:47:07.6214878Z 2022-11-23T02:47:07.6215129Z Running tests... 2022-11-23T02:47:07.6216271Z ---------------------------------------------------------------------- 2022-11-23T02:47:07.6219692Z test_grad_scaling (__main__.TestShardGradScaler) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.525s) 2022-11-23T02:47:07.6222442Z test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T02:47:07.6224901Z test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T02:47:07.6228923Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:07.6231988Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:07.6235333Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:07.6238452Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:07.6241884Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:07.6244923Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:07.6248002Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:07.6251084Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T02:47:07.6252346Z 2022-11-23T02:47:07.6253089Z ---------------------------------------------------------------------- 2022-11-23T02:47:07.6253950Z Ran 11 tests in 0.539s 2022-11-23T02:47:07.6254354Z 2022-11-23T02:47:07.6254603Z OK (skipped=11) 2022-11-23T02:47:07.6254983Z 2022-11-23T02:47:07.6255272Z Generating XML reports... 2022-11-23T02:47:07.6257049Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123024705.xml 2022-11-23T02:47:07.6259544Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123024705.xml 2022-11-23T02:47:07.6260701Z 2022-11-23T02:47:07.6261498Z ##[endgroup] 2022-11-23T02:47:07.6263296Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_lrjvycmd) 2022-11-23T02:47:07.6264314Z 2022-11-23T02:47:07.6265083Z Running distributed/fsdp/test_fsdp_overlap ... [2022-11-23 02:47:07.621477] 2022-11-23T02:47:07.6267193Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:47:07.622360] 2022-11-23T03:11:33.4030859Z 2022-11-23T03:11:33.4034880Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap 2022-11-23T03:11:33.4037793Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_overlap_ni3auq3a) 2022-11-23T03:11:33.4042594Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap 2022-11-23T03:11:33.4043350Z 2022-11-23T03:11:33.4046176Z Running tests... 2022-11-23T03:11:33.4048554Z ---------------------------------------------------------------------- 2022-11-23T03:11:33.4050145Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42387 2022-11-23T03:11:33.4052176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4053420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4055081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4056340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4057990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4059877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4061358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4062323Z dist init r=0, world=1 2022-11-23T03:11:33.4062710Z 2022-11-23T03:11:33.4062920Z rank0: 2022-11-23T03:11:33.4064289Z e1: {'cpu_iter': 0.0032083608999999846, 'cpu_wait': 4.664639999996112e-05, 'gpu_compute': 0.08553599966689944, 'gpu_total': 0.7834241986274719} 2022-11-23T03:11:33.4065927Z e2: {'cpu_iter': 0.010481364000000059, 'cpu_wait': 3.4654600000028066e-05, 'gpu_compute': 0.19742400078102945, 'gpu_total': 2.0292060017585754} 2022-11-23T03:11:33.4067554Z e3: {'cpu_iter': 0.004715688800000261, 'cpu_wait': 0.5880647762000002, 'gpu_compute': 339.37432250976565, 'gpu_total': 339.59324340820314} 2022-11-23T03:11:33.4069136Z e4: {'cpu_iter': 0.016413125100000057, 'cpu_wait': 0.5863092108000008, 'gpu_compute': 341.2075706481934, 'gpu_total': 341.741943359375} 2022-11-23T03:11:33.4072766Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4074995Z warnings.warn( 2022-11-23T03:11:33.4075608Z ok (27.464s) 2022-11-23T03:11:33.4077112Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 50 2022-11-23T03:11:33.4078508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42453 2022-11-23T03:11:33.4080953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4082133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4083733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4084992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4086202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4088034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4089695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4090636Z dist init r=0, world=1 2022-11-23T03:11:33.4091035Z 2022-11-23T03:11:33.4091244Z rank0: 2022-11-23T03:11:33.4092587Z e1: {'cpu_iter': 0.0033884343000000427, 'cpu_wait': 4.75551000000074e-05, 'gpu_compute': 0.0810240003746003, 'gpu_total': 0.8133283913135528} 2022-11-23T03:11:33.4094212Z e2: {'cpu_iter': 0.01130580520000013, 'cpu_wait': 3.826970000009311e-05, 'gpu_compute': 0.2285439983010292, 'gpu_total': 2.355865788459778} 2022-11-23T03:11:33.4095781Z e3: {'cpu_iter': 0.005423600699999742, 'cpu_wait': 0.5853640042, 'gpu_compute': 338.9033740997314, 'gpu_total': 339.1488006591797} 2022-11-23T03:11:33.4097331Z e4: {'cpu_iter': 0.01611235319999942, 'cpu_wait': 0.5861545443, 'gpu_compute': 341.240779876709, 'gpu_total': 341.6892425537109} 2022-11-23T03:11:33.4101623Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4103911Z warnings.warn( 2022-11-23T03:11:33.4104522Z ok (23.752s) 2022-11-23T03:11:33.4106002Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 49 2022-11-23T03:11:33.4107372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42519 2022-11-23T03:11:33.4109064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4110258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4111825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4113080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4114289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4116130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4117522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4118453Z dist init r=0, world=1 2022-11-23T03:11:33.4118862Z 2022-11-23T03:11:33.4119069Z rank0: 2022-11-23T03:11:33.4120544Z e1: {'cpu_iter': 0.002962734099999853, 'cpu_wait': 4.4375900000126475e-05, 'gpu_compute': 0.07543999976478517, 'gpu_total': 0.6952801942825317} 2022-11-23T03:11:33.4122176Z e2: {'cpu_iter': 0.010793140899999809, 'cpu_wait': 3.7187400000071594e-05, 'gpu_compute': 0.20185599885880948, 'gpu_total': 2.0754565954208375} 2022-11-23T03:11:33.4123746Z e3: {'cpu_iter': 0.005228581500000029, 'cpu_wait': 0.5906001547000006, 'gpu_compute': 340.95325927734376, 'gpu_total': 341.17841796875} 2022-11-23T03:11:33.4125308Z e4: {'cpu_iter': 0.015883489300000518, 'cpu_wait': 0.5823295445, 'gpu_compute': 339.0002807617187, 'gpu_total': 339.45066223144534} 2022-11-23T03:11:33.4128886Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4131006Z warnings.warn( 2022-11-23T03:11:33.4131600Z ok (33.286s) 2022-11-23T03:11:33.4133076Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 48 2022-11-23T03:11:33.4134603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42585 2022-11-23T03:11:33.4136326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4137511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4139095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4140353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4141562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4143385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4144794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4145741Z dist init r=0, world=1 2022-11-23T03:11:33.4146142Z 2022-11-23T03:11:33.4146344Z rank0: 2022-11-23T03:11:33.4147685Z e1: {'cpu_iter': 0.002354630399999902, 'cpu_wait': 3.8643900000145945e-05, 'gpu_compute': 0.045760000497102736, 'gpu_total': 0.5295520067214966} 2022-11-23T03:11:33.4149467Z e2: {'cpu_iter': 0.009223391199999842, 'cpu_wait': 2.882190000006446e-05, 'gpu_compute': 0.1666560009121895, 'gpu_total': 1.5818397998809814} 2022-11-23T03:11:33.4151079Z e3: {'cpu_iter': 0.004869614199999983, 'cpu_wait': 0.5811447189000001, 'gpu_compute': 336.24928245544436, 'gpu_total': 336.4742858886719} 2022-11-23T03:11:33.4152649Z e4: {'cpu_iter': 0.01635645700000197, 'cpu_wait': 0.5860724426999979, 'gpu_compute': 341.1323867797852, 'gpu_total': 341.5531433105469} 2022-11-23T03:11:33.4156227Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4158352Z warnings.warn( 2022-11-23T03:11:33.4158957Z ok (41.406s) 2022-11-23T03:11:33.4160549Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 47 2022-11-23T03:11:33.4161921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42651 2022-11-23T03:11:33.4163605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4164770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4166373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4167632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4168826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4170655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4172057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4172991Z dist init r=0, world=1 2022-11-23T03:11:33.4173389Z 2022-11-23T03:11:33.4173601Z rank0: 2022-11-23T03:11:33.4174922Z e1: {'cpu_iter': 0.0029099833000000076, 'cpu_wait': 4.160490000000294e-05, 'gpu_compute': 0.07656020000576973, 'gpu_total': 0.6688167989253998} 2022-11-23T03:11:33.4176531Z e2: {'cpu_iter': 0.010519788100000049, 'cpu_wait': 3.1540199999913196e-05, 'gpu_compute': 0.21702399905771017, 'gpu_total': 2.1491687893867493} 2022-11-23T03:11:33.4178241Z e3: {'cpu_iter': 0.005391304700000177, 'cpu_wait': 0.5884055485000002, 'gpu_compute': 340.2483184814453, 'gpu_total': 340.4750579833984} 2022-11-23T03:11:33.4179802Z e4: {'cpu_iter': 0.01562508290000082, 'cpu_wait': 0.5781608556999999, 'gpu_compute': 336.8271766662598, 'gpu_total': 337.24921875} 2022-11-23T03:11:33.4183353Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4185463Z warnings.warn( 2022-11-23T03:11:33.4186057Z ok (50.925s) 2022-11-23T03:11:33.4187532Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 46 2022-11-23T03:11:33.4188909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42717 2022-11-23T03:11:33.4190593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4191757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4193461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4194705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4195904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4197744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4199146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4200114Z dist init r=0, world=1 2022-11-23T03:11:33.4200609Z 2022-11-23T03:11:33.4200816Z rank0: 2022-11-23T03:11:33.4202153Z e1: {'cpu_iter': 0.0030239424000002126, 'cpu_wait': 4.539329999992958e-05, 'gpu_compute': 0.07841600002720953, 'gpu_total': 0.6900002002716065} 2022-11-23T03:11:33.4203652Z e2: {'cpu_iter': 0.010506737000000132, 'cpu_wait': 3.166929999993684e-05, 'gpu_compute': 0.22145599946379663, 'gpu_total': 2.176897406578064} 2022-11-23T03:11:33.4204278Z e3: {'cpu_iter': 0.005045384500000072, 'cpu_wait': 0.5884436891999979, 'gpu_compute': 339.8759117126465, 'gpu_total': 340.0986328125} 2022-11-23T03:11:33.4204894Z e4: {'cpu_iter': 0.015708299100001, 'cpu_wait': 0.5865338647999984, 'gpu_compute': 341.1056495666504, 'gpu_total': 341.5234436035156} 2022-11-23T03:11:33.4206307Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4207141Z warnings.warn( 2022-11-23T03:11:33.4207384Z ok (28.880s) 2022-11-23T03:11:33.4207973Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 45 2022-11-23T03:11:33.4208520Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42783 2022-11-23T03:11:33.4209187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4209662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4210293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4210847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4211323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4212045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4212601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4212970Z dist init r=0, world=1 2022-11-23T03:11:33.4213115Z 2022-11-23T03:11:33.4213201Z rank0: 2022-11-23T03:11:33.4213743Z e1: {'cpu_iter': 0.002675230000000095, 'cpu_wait': 4.161499999986162e-05, 'gpu_compute': 0.06839999984949827, 'gpu_total': 0.6299041986465455} 2022-11-23T03:11:33.4214389Z e2: {'cpu_iter': 0.010740678599999942, 'cpu_wait': 3.625720000002275e-05, 'gpu_compute': 0.218656000867486, 'gpu_total': 2.2246623992919923} 2022-11-23T03:11:33.4215015Z e3: {'cpu_iter': 0.0051457647000003, 'cpu_wait': 0.5797249330999985, 'gpu_compute': 335.94565162658694, 'gpu_total': 336.16608276367185} 2022-11-23T03:11:33.4215636Z e4: {'cpu_iter': 0.01363533630000191, 'cpu_wait': 0.5746143084000004, 'gpu_compute': 334.7220691680908, 'gpu_total': 335.15428466796874} 2022-11-23T03:11:33.4217078Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4217904Z warnings.warn( 2022-11-23T03:11:33.4218148Z ok (27.767s) 2022-11-23T03:11:33.4218737Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 44 2022-11-23T03:11:33.4219262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42849 2022-11-23T03:11:33.4219927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4220405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4221034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4221524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4222001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4222715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4223268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4223623Z dist init r=0, world=1 2022-11-23T03:11:33.4223785Z 2022-11-23T03:11:33.4223873Z rank0: 2022-11-23T03:11:33.4224409Z e1: {'cpu_iter': 0.0027914988000000028, 'cpu_wait': 4.2666000000046723e-05, 'gpu_compute': 0.08086400032043457, 'gpu_total': 0.7118510007858276} 2022-11-23T03:11:33.4225054Z e2: {'cpu_iter': 0.01053654230000003, 'cpu_wait': 3.0521299999897165e-05, 'gpu_compute': 0.1971522005274892, 'gpu_total': 2.056577181816101} 2022-11-23T03:11:33.4225681Z e3: {'cpu_iter': 0.0052369155000003605, 'cpu_wait': 0.5788260868999995, 'gpu_compute': 334.8460861206055, 'gpu_total': 335.12013244628906} 2022-11-23T03:11:33.4226307Z e4: {'cpu_iter': 0.014104261199999968, 'cpu_wait': 0.5763898842999999, 'gpu_compute': 335.6848134994507, 'gpu_total': 336.18320922851564} 2022-11-23T03:11:33.4227699Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4228578Z warnings.warn( 2022-11-23T03:11:33.4228822Z ok (21.153s) 2022-11-23T03:11:33.4229390Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 43 2022-11-23T03:11:33.4229939Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42915 2022-11-23T03:11:33.4230598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4231065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4231686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4232177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4232659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4233355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4233952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4234325Z dist init r=0, world=1 2022-11-23T03:11:33.4234484Z 2022-11-23T03:11:33.4234570Z rank0: 2022-11-23T03:11:33.4235101Z e1: {'cpu_iter': 0.002825823800000027, 'cpu_wait': 4.182180000009694e-05, 'gpu_compute': 0.06937600001692772, 'gpu_total': 0.6483043968677521} 2022-11-23T03:11:33.4235734Z e2: {'cpu_iter': 0.010742447000000155, 'cpu_wait': 3.580849999993418e-05, 'gpu_compute': 0.20942419953644276, 'gpu_total': 2.1182257771492004} 2022-11-23T03:11:33.4236360Z e3: {'cpu_iter': 0.005386979900000188, 'cpu_wait': 0.5794327693999994, 'gpu_compute': 335.75284519195554, 'gpu_total': 335.97462463378906} 2022-11-23T03:11:33.4236977Z e4: {'cpu_iter': 0.016359510300000223, 'cpu_wait': 0.5775328206999998, 'gpu_compute': 336.824471282959, 'gpu_total': 337.2554016113281} 2022-11-23T03:11:33.4238364Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4239194Z warnings.warn( 2022-11-23T03:11:33.4239425Z ok (18.150s) 2022-11-23T03:11:33.4240000Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 42 2022-11-23T03:11:33.4240598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42981 2022-11-23T03:11:33.4241267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4241739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4242365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4242852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4243325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4244025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4244581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4244947Z dist init r=0, world=1 2022-11-23T03:11:33.4245105Z 2022-11-23T03:11:33.4245191Z rank0: 2022-11-23T03:11:33.4245871Z e1: {'cpu_iter': 0.0033003717999999793, 'cpu_wait': 4.9098100000133374e-05, 'gpu_compute': 0.07363199992105365, 'gpu_total': 0.6850405968725681} 2022-11-23T03:11:33.4246515Z e2: {'cpu_iter': 0.011471020599999892, 'cpu_wait': 3.986470000008069e-05, 'gpu_compute': 0.25144040063023565, 'gpu_total': 2.475490427017212} 2022-11-23T03:11:33.4247144Z e3: {'cpu_iter': 0.005472258000000086, 'cpu_wait': 0.5902787754000001, 'gpu_compute': 341.05922775268556, 'gpu_total': 341.29205627441405} 2022-11-23T03:11:33.4247771Z e4: {'cpu_iter': 0.016169838099999723, 'cpu_wait': 0.5862482060000012, 'gpu_compute': 341.10873947143557, 'gpu_total': 341.57154541015626} 2022-11-23T03:11:33.4249158Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4249998Z warnings.warn( 2022-11-23T03:11:33.4250230Z ok (22.056s) 2022-11-23T03:11:33.4250906Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 41 2022-11-23T03:11:33.4251455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43047 2022-11-23T03:11:33.4252116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4252587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4253214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4253707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4254174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4254906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4255460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4255833Z dist init r=0, world=1 2022-11-23T03:11:33.4255992Z 2022-11-23T03:11:33.4256080Z rank0: 2022-11-23T03:11:33.4256611Z e1: {'cpu_iter': 0.0029393548999999907, 'cpu_wait': 4.461770000014908e-05, 'gpu_compute': 0.07372799972072244, 'gpu_total': 0.6549735963344574} 2022-11-23T03:11:33.4257258Z e2: {'cpu_iter': 0.010243167700000022, 'cpu_wait': 3.480389999985789e-05, 'gpu_compute': 0.18608039999380707, 'gpu_total': 1.9486090064048767} 2022-11-23T03:11:33.4257869Z e3: {'cpu_iter': 0.0052223257000002295, 'cpu_wait': 0.5756635827000002, 'gpu_compute': 333.97225694656373, 'gpu_total': 334.19810791015624} 2022-11-23T03:11:33.4258493Z e4: {'cpu_iter': 0.015725219399999803, 'cpu_wait': 0.5865827771999989, 'gpu_compute': 341.1167251586914, 'gpu_total': 341.5444641113281} 2022-11-23T03:11:33.4259887Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4260727Z warnings.warn( 2022-11-23T03:11:33.4260974Z ok (23.664s) 2022-11-23T03:11:33.4261563Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 40 2022-11-23T03:11:33.4262109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43113 2022-11-23T03:11:33.4262823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4263295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4263903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4264398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4264876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4265599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4266156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4266528Z dist init r=0, world=1 2022-11-23T03:11:33.4266690Z 2022-11-23T03:11:33.4266778Z rank0: 2022-11-23T03:11:33.4267304Z e1: {'cpu_iter': 0.002864153599999941, 'cpu_wait': 4.377250000020538e-05, 'gpu_compute': 0.07043199958279729, 'gpu_total': 0.6593129992485046} 2022-11-23T03:11:33.4267949Z e2: {'cpu_iter': 0.010412420100000031, 'cpu_wait': 3.1592699999905e-05, 'gpu_compute': 0.19689599936828017, 'gpu_total': 2.0805864095687867} 2022-11-23T03:11:33.4268627Z e3: {'cpu_iter': 0.0047872251999999445, 'cpu_wait': 0.5719740657000003, 'gpu_compute': 331.47961273193357, 'gpu_total': 331.70912780761716} 2022-11-23T03:11:33.4269254Z e4: {'cpu_iter': 0.015744645099999843, 'cpu_wait': 0.5864946535000005, 'gpu_compute': 334.76439571380615, 'gpu_total': 335.1877868652344} 2022-11-23T03:11:33.4270650Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4271477Z warnings.warn( 2022-11-23T03:11:33.4271720Z ok (25.069s) 2022-11-23T03:11:33.4272303Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 39 2022-11-23T03:11:33.4272846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43179 2022-11-23T03:11:33.4273505Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4273966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4274749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4275242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4275719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4276443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4276994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4277367Z dist init r=0, world=1 2022-11-23T03:11:33.4277524Z 2022-11-23T03:11:33.4277595Z rank0: 2022-11-23T03:11:33.4278122Z e1: {'cpu_iter': 0.002796657300000138, 'cpu_wait': 4.291260000011121e-05, 'gpu_compute': 0.07531200014054776, 'gpu_total': 0.7230087995529175} 2022-11-23T03:11:33.4278758Z e2: {'cpu_iter': 0.010618753399999913, 'cpu_wait': 3.549510000002698e-05, 'gpu_compute': 0.2080804008990526, 'gpu_total': 2.106210446357727} 2022-11-23T03:11:33.4279387Z e3: {'cpu_iter': 0.004497061500000043, 'cpu_wait': 0.5784112133000001, 'gpu_compute': 334.26033515930175, 'gpu_total': 334.50213317871095} 2022-11-23T03:11:33.4279994Z e4: {'cpu_iter': 0.015087426800000614, 'cpu_wait': 0.5861747543, 'gpu_compute': 339.9600997924805, 'gpu_total': 340.397021484375} 2022-11-23T03:11:33.4281505Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4282341Z warnings.warn( 2022-11-23T03:11:33.4282588Z ok (18.456s) 2022-11-23T03:11:33.4283173Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 38 2022-11-23T03:11:33.4283700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43245 2022-11-23T03:11:33.4284366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4284840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4285470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4286021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4286496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4287223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4287779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4288134Z dist init r=0, world=1 2022-11-23T03:11:33.4288291Z 2022-11-23T03:11:33.4288382Z rank0: 2022-11-23T03:11:33.4288914Z e1: {'cpu_iter': 0.0030603496999999534, 'cpu_wait': 4.492629999992559e-05, 'gpu_compute': 0.07379200020805002, 'gpu_total': 0.6919527947902679} 2022-11-23T03:11:33.4289554Z e2: {'cpu_iter': 0.010808390099999787, 'cpu_wait': 3.5824800000128223e-05, 'gpu_compute': 0.2088803995400667, 'gpu_total': 2.1149929761886597} 2022-11-23T03:11:33.4290187Z e3: {'cpu_iter': 0.005045587099999693, 'cpu_wait': 0.5907601789999998, 'gpu_compute': 340.87110290527346, 'gpu_total': 341.0896697998047} 2022-11-23T03:11:33.4290815Z e4: {'cpu_iter': 0.015883371499998588, 'cpu_wait': 0.5783721510000006, 'gpu_compute': 337.02306442260743, 'gpu_total': 337.47948303222654} 2022-11-23T03:11:33.4292196Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4293016Z warnings.warn( 2022-11-23T03:11:33.4293257Z ok (26.471s) 2022-11-23T03:11:33.4293821Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 37 2022-11-23T03:11:33.4294365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43311 2022-11-23T03:11:33.4295016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4295486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4296104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4296594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4297067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4297824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4298384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4298751Z dist init r=0, world=1 2022-11-23T03:11:33.4298911Z 2022-11-23T03:11:33.4299000Z rank0: 2022-11-23T03:11:33.4299521Z e1: {'cpu_iter': 0.0027493765, 'cpu_wait': 4.249259999982158e-05, 'gpu_compute': 0.07497619967907668, 'gpu_total': 0.6980805993080139} 2022-11-23T03:11:33.4300166Z e2: {'cpu_iter': 0.010764435099999936, 'cpu_wait': 3.664350000009975e-05, 'gpu_compute': 0.2186241988092661, 'gpu_total': 2.158976638317108} 2022-11-23T03:11:33.4300796Z e3: {'cpu_iter': 0.0047968099999999405, 'cpu_wait': 0.5830597202999996, 'gpu_compute': 337.7184341430664, 'gpu_total': 337.9384307861328} 2022-11-23T03:11:33.4301408Z e4: {'cpu_iter': 0.01553550039999969, 'cpu_wait': 0.5865758189000001, 'gpu_compute': 341.1762176513672, 'gpu_total': 341.6011444091797} 2022-11-23T03:11:33.4302841Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4303680Z warnings.warn( 2022-11-23T03:11:33.4303909Z ok (23.065s) 2022-11-23T03:11:33.4304485Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 36 2022-11-23T03:11:33.4305027Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43377 2022-11-23T03:11:33.4305685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4306159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4306784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4307275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4307744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4308453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4309002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4309370Z dist init r=0, world=1 2022-11-23T03:11:33.4309530Z 2022-11-23T03:11:33.4309619Z rank0: 2022-11-23T03:11:33.4310148Z e1: {'cpu_iter': 0.0029081706999999567, 'cpu_wait': 4.68386999999737e-05, 'gpu_compute': 0.07356820041313768, 'gpu_total': 0.6871208012104034} 2022-11-23T03:11:33.4310790Z e2: {'cpu_iter': 0.010427311799999917, 'cpu_wait': 3.116710000012901e-05, 'gpu_compute': 0.1953922001644969, 'gpu_total': 2.03475399017334} 2022-11-23T03:11:33.4311416Z e3: {'cpu_iter': 0.005449181499999689, 'cpu_wait': 0.5713725526999998, 'gpu_compute': 332.3858191490173, 'gpu_total': 332.6029052734375} 2022-11-23T03:11:33.4312026Z e4: {'cpu_iter': 0.017139618899999577, 'cpu_wait': 0.5732647161999992, 'gpu_compute': 335.03039207458494, 'gpu_total': 335.4803192138672} 2022-11-23T03:11:33.4313398Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4314264Z warnings.warn( 2022-11-23T03:11:33.4314507Z ok (22.265s) 2022-11-23T03:11:33.4315090Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 35 2022-11-23T03:11:33.4315629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43443 2022-11-23T03:11:33.4316294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4316765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4317387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4317860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4318342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4319058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4319610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4319981Z dist init r=0, world=1 2022-11-23T03:11:33.4320202Z 2022-11-23T03:11:33.4320289Z rank0: 2022-11-23T03:11:33.4320876Z e1: {'cpu_iter': 0.0030278848000000027, 'cpu_wait': 4.6373200000005e-05, 'gpu_compute': 0.0782082004006952, 'gpu_total': 0.7234653949737548} 2022-11-23T03:11:33.4321510Z e2: {'cpu_iter': 0.010629640699999766, 'cpu_wait': 3.6022099999932775e-05, 'gpu_compute': 0.19979199953377247, 'gpu_total': 2.0172067999839784} 2022-11-23T03:11:33.4322141Z e3: {'cpu_iter': 0.004709110499999891, 'cpu_wait': 0.5772702973000003, 'gpu_compute': 332.74291877746583, 'gpu_total': 332.9623291015625} 2022-11-23T03:11:33.4322761Z e4: {'cpu_iter': 0.015419113000000095, 'cpu_wait': 0.5779907340000008, 'gpu_compute': 336.8804344177246, 'gpu_total': 337.3105010986328} 2022-11-23T03:11:33.4324150Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4324986Z warnings.warn( 2022-11-23T03:11:33.4325231Z ok (17.755s) 2022-11-23T03:11:33.4325819Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 34 2022-11-23T03:11:33.4326368Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43509 2022-11-23T03:11:33.4327032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4327488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4328116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4328609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4329089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4329805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4330355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4330721Z dist init r=0, world=1 2022-11-23T03:11:33.4330882Z 2022-11-23T03:11:33.4330967Z rank0: 2022-11-23T03:11:33.4331481Z e1: {'cpu_iter': 0.003483725400000015, 'cpu_wait': 5.1845300000064046e-05, 'gpu_compute': 0.08993640011176467, 'gpu_total': 0.8410195887088776} 2022-11-23T03:11:33.4332124Z e2: {'cpu_iter': 0.011341254100000064, 'cpu_wait': 3.976510000001099e-05, 'gpu_compute': 0.2440488003194332, 'gpu_total': 2.395738422870636} 2022-11-23T03:11:33.4332805Z e3: {'cpu_iter': 0.005551544599999847, 'cpu_wait': 0.5876084317000004, 'gpu_compute': 339.9214668273926, 'gpu_total': 340.16343994140624} 2022-11-23T03:11:33.4333424Z e4: {'cpu_iter': 0.016844954099999754, 'cpu_wait': 0.5860652319999996, 'gpu_compute': 341.0999038696289, 'gpu_total': 341.56663513183594} 2022-11-23T03:11:33.4334812Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4335650Z warnings.warn( 2022-11-23T03:11:33.4335897Z ok (24.769s) 2022-11-23T03:11:33.4336474Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 33 2022-11-23T03:11:33.4337016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43575 2022-11-23T03:11:33.4337713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4338185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4338817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4339316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4339794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4340512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4341073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4341429Z dist init r=0, world=1 2022-11-23T03:11:33.4341590Z 2022-11-23T03:11:33.4341680Z rank0: 2022-11-23T03:11:33.4342219Z e1: {'cpu_iter': 0.002854704900000016, 'cpu_wait': 4.7298799999939686e-05, 'gpu_compute': 0.07094419994391502, 'gpu_total': 0.6932172000408172} 2022-11-23T03:11:33.4342860Z e2: {'cpu_iter': 0.01055641459999972, 'cpu_wait': 3.559650000015679e-05, 'gpu_compute': 0.19710759874433278, 'gpu_total': 2.033330035209656} 2022-11-23T03:11:33.4343497Z e3: {'cpu_iter': 0.005214841300000206, 'cpu_wait': 0.5783815335999997, 'gpu_compute': 335.21232318878174, 'gpu_total': 335.43556213378906} 2022-11-23T03:11:33.4344118Z e4: {'cpu_iter': 0.016016896400000036, 'cpu_wait': 0.5860531252000009, 'gpu_compute': 340.9619384765625, 'gpu_total': 341.3914276123047} 2022-11-23T03:11:33.4345508Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4346327Z warnings.warn( 2022-11-23T03:11:33.4346575Z ok (25.371s) 2022-11-23T03:11:33.4347148Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 32 2022-11-23T03:11:33.4347694Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43641 2022-11-23T03:11:33.4348357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4348831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4349508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4350002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4350485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4351212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4351750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4352115Z dist init r=0, world=1 2022-11-23T03:11:33.4352275Z 2022-11-23T03:11:33.4352363Z rank0: 2022-11-23T03:11:33.4352894Z e1: {'cpu_iter': 0.0030064407999997656, 'cpu_wait': 4.63725000000359e-05, 'gpu_compute': 0.06145599959418178, 'gpu_total': 0.6304647982120514} 2022-11-23T03:11:33.4353534Z e2: {'cpu_iter': 0.010664349900000225, 'cpu_wait': 3.6542499999825394e-05, 'gpu_compute': 0.20126419886946678, 'gpu_total': 2.110959994792938} 2022-11-23T03:11:33.4354168Z e3: {'cpu_iter': 0.005012447599999881, 'cpu_wait': 0.5772474901999993, 'gpu_compute': 334.5450569152832, 'gpu_total': 334.76629638671875} 2022-11-23T03:11:33.4354845Z e4: {'cpu_iter': 0.016123917899999184, 'cpu_wait': 0.5739051730000007, 'gpu_compute': 334.84334564208984, 'gpu_total': 335.26930541992186} 2022-11-23T03:11:33.4356247Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4357086Z warnings.warn( 2022-11-23T03:11:33.4357315Z ok (21.061s) 2022-11-23T03:11:33.4357896Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 31 2022-11-23T03:11:33.4358437Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43707 2022-11-23T03:11:33.4359105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4359582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4360250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4360746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4361227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4361931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4362492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4362863Z dist init r=0, world=1 2022-11-23T03:11:33.4363023Z 2022-11-23T03:11:33.4363112Z rank0: 2022-11-23T03:11:33.4363646Z e1: {'cpu_iter': 0.002784561900000071, 'cpu_wait': 4.364710000013261e-05, 'gpu_compute': 0.07593600014224648, 'gpu_total': 0.6617768108844757} 2022-11-23T03:11:33.4364281Z e2: {'cpu_iter': 0.010570662700000089, 'cpu_wait': 3.690920000005704e-05, 'gpu_compute': 0.20811620093882083, 'gpu_total': 2.0872525930404664} 2022-11-23T03:11:33.4364906Z e3: {'cpu_iter': 0.004896642400000317, 'cpu_wait': 0.5783793394000002, 'gpu_compute': 334.90285415649413, 'gpu_total': 335.12586364746096} 2022-11-23T03:11:33.4365510Z e4: {'cpu_iter': 0.016341370300000692, 'cpu_wait': 0.5858522208000003, 'gpu_compute': 341.0857963562012, 'gpu_total': 341.52203369140625} 2022-11-23T03:11:33.4366896Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4367784Z warnings.warn( 2022-11-23T03:11:33.4368028Z ok (22.364s) 2022-11-23T03:11:33.4368604Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 30 2022-11-23T03:11:33.4369146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43773 2022-11-23T03:11:33.4369808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4370279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4370901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4371385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4371862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4372631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4373187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4373552Z dist init r=0, world=1 2022-11-23T03:11:33.4373711Z 2022-11-23T03:11:33.4373798Z rank0: 2022-11-23T03:11:33.4374327Z e1: {'cpu_iter': 0.003034976899999897, 'cpu_wait': 4.5159900000069085e-05, 'gpu_compute': 0.08627200024202467, 'gpu_total': 0.7667685985565186} 2022-11-23T03:11:33.4374964Z e2: {'cpu_iter': 0.010689940999999958, 'cpu_wait': 3.65861000000578e-05, 'gpu_compute': 0.21411239951848984, 'gpu_total': 2.1759171962738035} 2022-11-23T03:11:33.4375639Z e3: {'cpu_iter': 0.005088020699999607, 'cpu_wait': 0.5902629931999999, 'gpu_compute': 341.0721038818359, 'gpu_total': 341.291015625} 2022-11-23T03:11:33.4376406Z e4: {'cpu_iter': 0.01592744070000087, 'cpu_wait': 0.5862211807000005, 'gpu_compute': 341.1235977172852, 'gpu_total': 341.5472442626953} 2022-11-23T03:11:33.4377796Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4378635Z warnings.warn( 2022-11-23T03:11:33.4378878Z ok (24.769s) 2022-11-23T03:11:33.4379453Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 29 2022-11-23T03:11:33.4380001Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43839 2022-11-23T03:11:33.4380659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4381134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4381752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4382243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4382721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4383438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4383991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4384538Z dist init r=0, world=1 2022-11-23T03:11:33.4384704Z 2022-11-23T03:11:33.4384796Z rank0: 2022-11-23T03:11:33.4385320Z e1: {'cpu_iter': 0.0027862663999999706, 'cpu_wait': 4.0995799999876684e-05, 'gpu_compute': 0.06999999992549419, 'gpu_total': 0.6365285992622376} 2022-11-23T03:11:33.4385976Z e2: {'cpu_iter': 0.010486948000000051, 'cpu_wait': 3.329729999981268e-05, 'gpu_compute': 0.21131199868395925, 'gpu_total': 2.192177414894104} 2022-11-23T03:11:33.4386602Z e3: {'cpu_iter': 0.005288368200000093, 'cpu_wait': 0.5903889043000001, 'gpu_compute': 340.93861083984376, 'gpu_total': 341.15977478027344} 2022-11-23T03:11:33.4387225Z e4: {'cpu_iter': 0.016310114600000248, 'cpu_wait': 0.5857628811999998, 'gpu_compute': 340.9956214904785, 'gpu_total': 341.4216033935547} 2022-11-23T03:11:33.4388616Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4389452Z warnings.warn( 2022-11-23T03:11:33.4389743Z ok (22.965s) 2022-11-23T03:11:33.4390331Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 28 2022-11-23T03:11:33.4390871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43905 2022-11-23T03:11:33.4391519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4391993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4392616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4393115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4393593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4394316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4394870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4395234Z dist init r=0, world=1 2022-11-23T03:11:33.4395378Z 2022-11-23T03:11:33.4395466Z rank0: 2022-11-23T03:11:33.4395999Z e1: {'cpu_iter': 0.0028772622000001746, 'cpu_wait': 4.4671000000029436e-05, 'gpu_compute': 0.07478400012478233, 'gpu_total': 0.7176330029964447} 2022-11-23T03:11:33.4396643Z e2: {'cpu_iter': 0.010579966699999943, 'cpu_wait': 3.618920000025838e-05, 'gpu_compute': 0.20454440042376518, 'gpu_total': 2.087489402294159} 2022-11-23T03:11:33.4397267Z e3: {'cpu_iter': 0.004501626199999898, 'cpu_wait': 0.5697717562000004, 'gpu_compute': 330.2646718978882, 'gpu_total': 330.4965759277344} 2022-11-23T03:11:33.4397884Z e4: {'cpu_iter': 0.01481780340000043, 'cpu_wait': 0.5861419749999996, 'gpu_compute': 340.80204849243165, 'gpu_total': 341.2228668212891} 2022-11-23T03:11:33.4399279Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4400092Z warnings.warn( 2022-11-23T03:11:33.4400389Z ok (17.754s) 2022-11-23T03:11:33.4400972Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 27 2022-11-23T03:11:33.4401565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43971 2022-11-23T03:11:33.4402229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4402706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4403325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4403812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4404288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4405004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4405566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4405927Z dist init r=0, world=1 2022-11-23T03:11:33.4406086Z 2022-11-23T03:11:33.4406175Z rank0: 2022-11-23T03:11:33.4406707Z e1: {'cpu_iter': 0.002968919799999892, 'cpu_wait': 4.5642700000136924e-05, 'gpu_compute': 0.07403220003470778, 'gpu_total': 0.7052170097827911} 2022-11-23T03:11:33.4407399Z e2: {'cpu_iter': 0.010475627800000087, 'cpu_wait': 3.678659999986067e-05, 'gpu_compute': 0.2162240006029606, 'gpu_total': 2.161177027225494} 2022-11-23T03:11:33.4408028Z e3: {'cpu_iter': 0.00499289079999965, 'cpu_wait': 0.5720005332000005, 'gpu_compute': 331.77187938690184, 'gpu_total': 332.0100372314453} 2022-11-23T03:11:33.4408641Z e4: {'cpu_iter': 0.016205616399999557, 'cpu_wait': 0.5858672587000008, 'gpu_compute': 341.100545501709, 'gpu_total': 341.5545959472656} 2022-11-23T03:11:33.4410026Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4410859Z warnings.warn( 2022-11-23T03:11:33.4411104Z ok (30.380s) 2022-11-23T03:11:33.4411670Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 26 2022-11-23T03:11:33.4412207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44037 2022-11-23T03:11:33.4412867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4413338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4413960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4414461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4414938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4415638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4416196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4416562Z dist init r=0, world=1 2022-11-23T03:11:33.4416722Z 2022-11-23T03:11:33.4416810Z rank0: 2022-11-23T03:11:33.4417336Z e1: {'cpu_iter': 0.0030641278999999242, 'cpu_wait': 4.794139999999558e-05, 'gpu_compute': 0.07535999985411763, 'gpu_total': 0.7230326056480407} 2022-11-23T03:11:33.4417980Z e2: {'cpu_iter': 0.010509134599999914, 'cpu_wait': 3.645310000015556e-05, 'gpu_compute': 0.21206419877707958, 'gpu_total': 2.1233303904533387} 2022-11-23T03:11:33.4418598Z e3: {'cpu_iter': 0.00537309999999982, 'cpu_wait': 0.5736929669000003, 'gpu_compute': 332.58649921417236, 'gpu_total': 332.8763000488281} 2022-11-23T03:11:33.4446667Z e4: {'cpu_iter': 0.016262346200000267, 'cpu_wait': 0.5863983180999991, 'gpu_compute': 335.941227722168, 'gpu_total': 336.36639709472655} 2022-11-23T03:11:33.4448129Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4448989Z warnings.warn( 2022-11-23T03:11:33.4449239Z ok (19.056s) 2022-11-23T03:11:33.4449813Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 25 2022-11-23T03:11:33.4450363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44103 2022-11-23T03:11:33.4451027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4451493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4452303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4452800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4453272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4453977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4454529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4454896Z dist init r=0, world=1 2022-11-23T03:11:33.4455061Z 2022-11-23T03:11:33.4455148Z rank0: 2022-11-23T03:11:33.4455680Z e1: {'cpu_iter': 0.0029634191000001307, 'cpu_wait': 4.50122000001052e-05, 'gpu_compute': 0.07817620029672981, 'gpu_total': 0.7052896022796631} 2022-11-23T03:11:33.4456330Z e2: {'cpu_iter': 0.010535378099999982, 'cpu_wait': 3.6232299999916505e-05, 'gpu_compute': 0.20172819988802076, 'gpu_total': 2.1333693861961365} 2022-11-23T03:11:33.4456961Z e3: {'cpu_iter': 0.004870636400000361, 'cpu_wait': 0.5716120575999998, 'gpu_compute': 331.5541658401489, 'gpu_total': 331.77362365722655} 2022-11-23T03:11:33.4457575Z e4: {'cpu_iter': 0.015699567899999777, 'cpu_wait': 0.5857365511000012, 'gpu_compute': 338.77783355712893, 'gpu_total': 339.2051696777344} 2022-11-23T03:11:33.4458953Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4459780Z warnings.warn( 2022-11-23T03:11:33.4460009Z ok (17.755s) 2022-11-23T03:11:33.4460582Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 24 2022-11-23T03:11:33.4461121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44169 2022-11-23T03:11:33.4461778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4462242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4462861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4463348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4463894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4464603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4465150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4465513Z dist init r=0, world=1 2022-11-23T03:11:33.4465672Z 2022-11-23T03:11:33.4465757Z rank0: 2022-11-23T03:11:33.4466282Z e1: {'cpu_iter': 0.002848262600000062, 'cpu_wait': 4.296670000005776e-05, 'gpu_compute': 0.0804482001811266, 'gpu_total': 0.7108807981014251} 2022-11-23T03:11:33.4466926Z e2: {'cpu_iter': 0.010511637999999924, 'cpu_wait': 3.1447000000017766e-05, 'gpu_compute': 0.21457639746367932, 'gpu_total': 2.1458903789520263} 2022-11-23T03:11:33.4467534Z e3: {'cpu_iter': 0.005147930799999667, 'cpu_wait': 0.5841369030999999, 'gpu_compute': 338.11168518066404, 'gpu_total': 338.33216857910156} 2022-11-23T03:11:33.4468155Z e4: {'cpu_iter': 0.01562608739999973, 'cpu_wait': 0.5864462879999991, 'gpu_compute': 341.15785827636716, 'gpu_total': 341.58172607421875} 2022-11-23T03:11:33.4469617Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4470438Z warnings.warn( 2022-11-23T03:11:33.4470680Z ok (27.776s) 2022-11-23T03:11:33.4471249Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 23 2022-11-23T03:11:33.4471779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44235 2022-11-23T03:11:33.4472434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4472894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4473499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4473974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4474444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4475153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4475683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4476041Z dist init r=0, world=1 2022-11-23T03:11:33.4476198Z 2022-11-23T03:11:33.4476279Z rank0: 2022-11-23T03:11:33.4476794Z e1: {'cpu_iter': 0.0029392582999999027, 'cpu_wait': 4.331720000019246e-05, 'gpu_compute': 0.06670399997383356, 'gpu_total': 0.6720184028148651} 2022-11-23T03:11:33.4477432Z e2: {'cpu_iter': 0.010341293800000173, 'cpu_wait': 3.1060199999988214e-05, 'gpu_compute': 0.1949286000803113, 'gpu_total': 2.0326441884040833} 2022-11-23T03:11:33.4478201Z e3: {'cpu_iter': 0.0048360828000001636, 'cpu_wait': 0.5873187710999995, 'gpu_compute': 339.60350189208987, 'gpu_total': 339.8203979492188} 2022-11-23T03:11:33.4478817Z e4: {'cpu_iter': 0.015509927700001213, 'cpu_wait': 0.5781121109000005, 'gpu_compute': 336.93718948364256, 'gpu_total': 337.36366577148436} 2022-11-23T03:11:33.4480262Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4481142Z warnings.warn( 2022-11-23T03:11:33.4481376Z ok (24.368s) 2022-11-23T03:11:33.4481957Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 22 2022-11-23T03:11:33.4482492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44301 2022-11-23T03:11:33.4483144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4483602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4484238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4484717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4485195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4485907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4486504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4486857Z dist init r=0, world=1 2022-11-23T03:11:33.4487016Z 2022-11-23T03:11:33.4487088Z rank0: 2022-11-23T03:11:33.4487610Z e1: {'cpu_iter': 0.00284959580000006, 'cpu_wait': 4.2940899999877046e-05, 'gpu_compute': 0.07632019910961389, 'gpu_total': 0.6836807906627655} 2022-11-23T03:11:33.4488238Z e2: {'cpu_iter': 0.010549676000000119, 'cpu_wait': 3.1818099999991746e-05, 'gpu_compute': 0.21048020161688327, 'gpu_total': 2.088060998916626} 2022-11-23T03:11:33.4488847Z e3: {'cpu_iter': 0.005289327399999611, 'cpu_wait': 0.5840280432999998, 'gpu_compute': 338.0878505706787, 'gpu_total': 338.3155090332031} 2022-11-23T03:11:33.4489450Z e4: {'cpu_iter': 0.015609874899999454, 'cpu_wait': 0.586830034299999, 'gpu_compute': 341.1044792175293, 'gpu_total': 341.5323486328125} 2022-11-23T03:11:33.4490831Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4491644Z warnings.warn( 2022-11-23T03:11:33.4491879Z ok (27.073s) 2022-11-23T03:11:33.4492444Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 21 2022-11-23T03:11:33.4492979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44367 2022-11-23T03:11:33.4493618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4494080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4494697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4495177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4495641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4496341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4496879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4497228Z dist init r=0, world=1 2022-11-23T03:11:33.4497380Z 2022-11-23T03:11:33.4497461Z rank0: 2022-11-23T03:11:33.4498039Z e1: {'cpu_iter': 0.0029550867999999396, 'cpu_wait': 4.387530000000694e-05, 'gpu_compute': 0.07104020034894347, 'gpu_total': 0.6617936015129089} 2022-11-23T03:11:33.4498667Z e2: {'cpu_iter': 0.010488097099999916, 'cpu_wait': 3.2217300000070284e-05, 'gpu_compute': 0.200464397855103, 'gpu_total': 2.0675071716308593} 2022-11-23T03:11:33.4499284Z e3: {'cpu_iter': 0.005377349699999456, 'cpu_wait': 0.5775265691999998, 'gpu_compute': 334.8040609359741, 'gpu_total': 335.02272338867186} 2022-11-23T03:11:33.4499875Z e4: {'cpu_iter': 0.015932682800000108, 'cpu_wait': 0.5743350868, 'gpu_compute': 335.107843208313, 'gpu_total': 335.528076171875} 2022-11-23T03:11:33.4501249Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4502054Z warnings.warn( 2022-11-23T03:11:33.4502288Z ok (19.056s) 2022-11-23T03:11:33.4502846Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 20 2022-11-23T03:11:33.4503429Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44433 2022-11-23T03:11:33.4504088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4504542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4505152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4505633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4506100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4506811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4507352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4507712Z dist init r=0, world=1 2022-11-23T03:11:33.4507863Z 2022-11-23T03:11:33.4507946Z rank0: 2022-11-23T03:11:33.4508461Z e1: {'cpu_iter': 0.0029122158999999926, 'cpu_wait': 4.2618799999871725e-05, 'gpu_compute': 0.07265620017424226, 'gpu_total': 0.6576175928115845} 2022-11-23T03:11:33.4509092Z e2: {'cpu_iter': 0.010388354199999927, 'cpu_wait': 3.115549999970213e-05, 'gpu_compute': 0.19323219936341046, 'gpu_total': 2.060059976577759} 2022-11-23T03:11:33.4509711Z e3: {'cpu_iter': 0.005239095800000282, 'cpu_wait': 0.5717320894000004, 'gpu_compute': 331.6390684127808, 'gpu_total': 331.88096923828124} 2022-11-23T03:11:33.4510312Z e4: {'cpu_iter': 0.015746144299999897, 'cpu_wait': 0.579894663599999, 'gpu_compute': 337.7007373809814, 'gpu_total': 338.12916259765626} 2022-11-23T03:11:33.4511698Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4512523Z warnings.warn( 2022-11-23T03:11:33.4512745Z ok (21.263s) 2022-11-23T03:11:33.4513306Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 19 2022-11-23T03:11:33.4513837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44499 2022-11-23T03:11:33.4514481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4514993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4515606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4516095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4516552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4517251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4517788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4518142Z dist init r=0, world=1 2022-11-23T03:11:33.4518297Z 2022-11-23T03:11:33.4518375Z rank0: 2022-11-23T03:11:33.4518904Z e1: {'cpu_iter': 0.0028061998000000975, 'cpu_wait': 4.4153400000013224e-05, 'gpu_compute': 0.07348799956962467, 'gpu_total': 0.6460006058216095} 2022-11-23T03:11:33.4519531Z e2: {'cpu_iter': 0.010162484599999999, 'cpu_wait': 3.106690000000967e-05, 'gpu_compute': 0.1892942011356354, 'gpu_total': 2.001521384716034} 2022-11-23T03:11:33.4520243Z e3: {'cpu_iter': 0.004379591300000296, 'cpu_wait': 0.5847970503000003, 'gpu_compute': 337.80455017089844, 'gpu_total': 338.02398986816405} 2022-11-23T03:11:33.4520842Z e4: {'cpu_iter': 0.0137853726000003, 'cpu_wait': 0.5871889227000004, 'gpu_compute': 341.1691864013672, 'gpu_total': 341.59620666503906} 2022-11-23T03:11:33.4522214Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4523035Z warnings.warn( 2022-11-23T03:11:33.4523265Z ok (24.268s) 2022-11-23T03:11:33.4523834Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 18 2022-11-23T03:11:33.4524369Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44565 2022-11-23T03:11:33.4525014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4525473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4526088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4526568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4527036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4527745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4528292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4528644Z dist init r=0, world=1 2022-11-23T03:11:33.4528802Z 2022-11-23T03:11:33.4528885Z rank0: 2022-11-23T03:11:33.4529400Z e1: {'cpu_iter': 0.002926244700000069, 'cpu_wait': 3.925330000003058e-05, 'gpu_compute': 0.08065600027330219, 'gpu_total': 0.71771320104599} 2022-11-23T03:11:33.4530018Z e2: {'cpu_iter': 0.010441051600000062, 'cpu_wait': 3.102589999990357e-05, 'gpu_compute': 0.2037605991587043, 'gpu_total': 2.044691801071167} 2022-11-23T03:11:33.4530639Z e3: {'cpu_iter': 0.005095805499999884, 'cpu_wait': 0.5882949588999999, 'gpu_compute': 340.1360137939453, 'gpu_total': 340.3544189453125} 2022-11-23T03:11:33.4531244Z e4: {'cpu_iter': 0.015949066100000308, 'cpu_wait': 0.5790664716999991, 'gpu_compute': 337.57630767822263, 'gpu_total': 338.0102783203125} 2022-11-23T03:11:33.4532676Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4533498Z warnings.warn( 2022-11-23T03:11:33.4533726Z ok (20.159s) 2022-11-23T03:11:33.4534293Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 17 2022-11-23T03:11:33.4534825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44631 2022-11-23T03:11:33.4535472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4535927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4536542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4537015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4537565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4538277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4538820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4539174Z dist init r=0, world=1 2022-11-23T03:11:33.4539331Z 2022-11-23T03:11:33.4539409Z rank0: 2022-11-23T03:11:33.4539920Z e1: {'cpu_iter': 0.002706087700000026, 'cpu_wait': 4.372119999986879e-05, 'gpu_compute': 0.06851200037635863, 'gpu_total': 0.6369772017002105} 2022-11-23T03:11:33.4540546Z e2: {'cpu_iter': 0.010197742100000263, 'cpu_wait': 3.1404199999940376e-05, 'gpu_compute': 0.20534439999610185, 'gpu_total': 2.0582273960113526} 2022-11-23T03:11:33.4541167Z e3: {'cpu_iter': 0.004791845799999805, 'cpu_wait': 0.5693086768000004, 'gpu_compute': 330.4850685119629, 'gpu_total': 330.71266479492186} 2022-11-23T03:11:33.4541773Z e4: {'cpu_iter': 0.01562161169999996, 'cpu_wait': 0.5717047369999996, 'gpu_compute': 333.861646270752, 'gpu_total': 334.28997497558595} 2022-11-23T03:11:33.4543149Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4543960Z warnings.warn( 2022-11-23T03:11:33.4544195Z ok (17.654s) 2022-11-23T03:11:33.4544760Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 16 2022-11-23T03:11:33.4545290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44697 2022-11-23T03:11:33.4545930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4546396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4547001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4547479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4547943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4548649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4549243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4549585Z dist init r=0, world=1 2022-11-23T03:11:33.4549736Z 2022-11-23T03:11:33.4549819Z rank0: 2022-11-23T03:11:33.4550338Z e1: {'cpu_iter': 0.002953298699999962, 'cpu_wait': 4.454849999975252e-05, 'gpu_compute': 0.07179199997335672, 'gpu_total': 0.6390087962150574} 2022-11-23T03:11:33.4550969Z e2: {'cpu_iter': 0.010698306599999974, 'cpu_wait': 3.1728899999894365e-05, 'gpu_compute': 0.22457619942724705, 'gpu_total': 2.2152187824249268} 2022-11-23T03:11:33.4551571Z e3: {'cpu_iter': 0.005289176200000423, 'cpu_wait': 0.5882955673999982, 'gpu_compute': 339.877725982666, 'gpu_total': 340.0993347167969} 2022-11-23T03:11:33.4552177Z e4: {'cpu_iter': 0.015886652500000053, 'cpu_wait': 0.5865436096000003, 'gpu_compute': 341.12993240356445, 'gpu_total': 341.5438537597656} 2022-11-23T03:11:33.4553600Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4554404Z warnings.warn( 2022-11-23T03:11:33.4554638Z ok (34.691s) 2022-11-23T03:11:33.4555204Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 15 2022-11-23T03:11:33.4555733Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44763 2022-11-23T03:11:33.4556376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4556827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4557436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4557913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4558379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4559085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4559614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4559969Z dist init r=0, world=1 2022-11-23T03:11:33.4560123Z 2022-11-23T03:11:33.4560263Z rank0: 2022-11-23T03:11:33.4560783Z e1: {'cpu_iter': 0.0028479486000000166, 'cpu_wait': 4.614239999991554e-05, 'gpu_compute': 0.0763361994177103, 'gpu_total': 0.7058081984519958} 2022-11-23T03:11:33.4561421Z e2: {'cpu_iter': 0.010199539300000105, 'cpu_wait': 3.151939999979092e-05, 'gpu_compute': 0.1925765985623002, 'gpu_total': 1.9873307585716247} 2022-11-23T03:11:33.4562019Z e3: {'cpu_iter': 0.005257237599999609, 'cpu_wait': 0.5787663884, 'gpu_compute': 335.6010271072388, 'gpu_total': 335.81727294921876} 2022-11-23T03:11:33.4562625Z e4: {'cpu_iter': 0.01568658520000028, 'cpu_wait': 0.5861751777000002, 'gpu_compute': 341.1465606689453, 'gpu_total': 341.5668884277344} 2022-11-23T03:11:33.4564007Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4564822Z warnings.warn( 2022-11-23T03:11:33.4565108Z ok (19.256s) 2022-11-23T03:11:33.4565673Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 14 2022-11-23T03:11:33.4566197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44829 2022-11-23T03:11:33.4566847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4567305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4567922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4568408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4568875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4569568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4570110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4570468Z dist init r=0, world=1 2022-11-23T03:11:33.4570616Z 2022-11-23T03:11:33.4570700Z rank0: 2022-11-23T03:11:33.4571266Z e1: {'cpu_iter': 0.0030264489999999532, 'cpu_wait': 4.508540000003336e-05, 'gpu_compute': 0.07804840002208949, 'gpu_total': 0.6809927999973298} 2022-11-23T03:11:33.4571902Z e2: {'cpu_iter': 0.010580554299999979, 'cpu_wait': 3.1311999999950046e-05, 'gpu_compute': 0.20668819919228554, 'gpu_total': 2.1044819951057434} 2022-11-23T03:11:33.4572516Z e3: {'cpu_iter': 0.0052567801999996, 'cpu_wait': 0.5898150980000002, 'gpu_compute': 340.6662452697754, 'gpu_total': 340.8841613769531} 2022-11-23T03:11:33.4573113Z e4: {'cpu_iter': 0.016040085899999924, 'cpu_wait': 0.5859722797999997, 'gpu_compute': 340.69583587646486, 'gpu_total': 341.12575073242186} 2022-11-23T03:11:33.4574493Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4575463Z warnings.warn( 2022-11-23T03:11:33.4575690Z ok (22.660s) 2022-11-23T03:11:33.4576253Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 13 2022-11-23T03:11:33.4576793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44895 2022-11-23T03:11:33.4577439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4577901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4578525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4578994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4579544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4580251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4580786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4581141Z dist init r=0, world=1 2022-11-23T03:11:33.4581298Z 2022-11-23T03:11:33.4581382Z rank0: 2022-11-23T03:11:33.4581907Z e1: {'cpu_iter': 0.002891303900000164, 'cpu_wait': 4.308820000025193e-05, 'gpu_compute': 0.07742419959977269, 'gpu_total': 0.6840971946716309} 2022-11-23T03:11:33.4582548Z e2: {'cpu_iter': 0.010550937500000046, 'cpu_wait': 3.159870000013498e-05, 'gpu_compute': 0.22212859913706778, 'gpu_total': 2.1740352272987367} 2022-11-23T03:11:33.4583212Z e3: {'cpu_iter': 0.004276767899999978, 'cpu_wait': 0.5878643664000001, 'gpu_compute': 339.6675086975098, 'gpu_total': 339.8893341064453} 2022-11-23T03:11:33.4583823Z e4: {'cpu_iter': 0.015886051499999353, 'cpu_wait': 0.5808442241999998, 'gpu_compute': 338.38026695251466, 'gpu_total': 338.80230102539065} 2022-11-23T03:11:33.4585204Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4586011Z warnings.warn( 2022-11-23T03:11:33.4586250Z ok (19.454s) 2022-11-23T03:11:33.4586825Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 12 2022-11-23T03:11:33.4587357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44961 2022-11-23T03:11:33.4588059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4588524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4589127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4589610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4590084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4590802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4591355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4591718Z dist init r=0, world=1 2022-11-23T03:11:33.4591874Z 2022-11-23T03:11:33.4591959Z rank0: 2022-11-23T03:11:33.4592477Z e1: {'cpu_iter': 0.002904558600000051, 'cpu_wait': 4.372529999994157e-05, 'gpu_compute': 0.07064019972458482, 'gpu_total': 0.6404168009757996} 2022-11-23T03:11:33.4593117Z e2: {'cpu_iter': 0.010471291300000019, 'cpu_wait': 3.1094599999992314e-05, 'gpu_compute': 0.19859260078519583, 'gpu_total': 2.0768407940864564} 2022-11-23T03:11:33.4593734Z e3: {'cpu_iter': 0.00522961910000026, 'cpu_wait': 0.5900708442999993, 'gpu_compute': 341.08508987426757, 'gpu_total': 341.3047760009766} 2022-11-23T03:11:33.4594344Z e4: {'cpu_iter': 0.015674074000000003, 'cpu_wait': 0.5776854355000012, 'gpu_compute': 336.5243404388428, 'gpu_total': 336.9485321044922} 2022-11-23T03:11:33.4595728Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4596571Z warnings.warn( 2022-11-23T03:11:33.4596811Z ok (25.672s) 2022-11-23T03:11:33.4597386Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 11 2022-11-23T03:11:33.4597921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45027 2022-11-23T03:11:33.4598569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4599021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4599643Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4600237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4600716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4601439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4601985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4602350Z dist init r=0, world=1 2022-11-23T03:11:33.4602494Z 2022-11-23T03:11:33.4602581Z rank0: 2022-11-23T03:11:33.4603112Z e1: {'cpu_iter': 0.002792465699999891, 'cpu_wait': 4.1691200000215646e-05, 'gpu_compute': 0.0729441998526454, 'gpu_total': 0.6584808051586151} 2022-11-23T03:11:33.4603751Z e2: {'cpu_iter': 0.010455607399999955, 'cpu_wait': 3.5589500000110516e-05, 'gpu_compute': 0.21225619949400426, 'gpu_total': 2.1229950308799745} 2022-11-23T03:11:33.4604375Z e3: {'cpu_iter': 0.005911077299999956, 'cpu_wait': 0.5899581687999997, 'gpu_compute': 341.1777648925781, 'gpu_total': 341.43329467773435} 2022-11-23T03:11:33.4605047Z e4: {'cpu_iter': 0.016581531199999944, 'cpu_wait': 0.5861858480000016, 'gpu_compute': 341.23049240112306, 'gpu_total': 341.7312042236328} 2022-11-23T03:11:33.4606428Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4607261Z warnings.warn( 2022-11-23T03:11:33.4607499Z ok (27.176s) 2022-11-23T03:11:33.4608069Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 10 2022-11-23T03:11:33.4608593Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45093 2022-11-23T03:11:33.4609245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4609713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4610333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4610828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4611307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4612012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4612559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4612916Z dist init r=0, world=1 2022-11-23T03:11:33.4613071Z 2022-11-23T03:11:33.4613161Z rank0: 2022-11-23T03:11:33.4613687Z e1: {'cpu_iter': 0.003000228199999899, 'cpu_wait': 4.244319999990864e-05, 'gpu_compute': 0.08956799991428852, 'gpu_total': 0.7722563922405243} 2022-11-23T03:11:33.4614326Z e2: {'cpu_iter': 0.010505467200000052, 'cpu_wait': 3.2582600000097275e-05, 'gpu_compute': 0.20035239979624747, 'gpu_total': 2.0662104368209837} 2022-11-23T03:11:33.4614948Z e3: {'cpu_iter': 0.005186652500000033, 'cpu_wait': 0.5755062179999998, 'gpu_compute': 333.5550106048584, 'gpu_total': 333.79315490722655} 2022-11-23T03:11:33.4615564Z e4: {'cpu_iter': 0.015144572500000208, 'cpu_wait': 0.5864809977000001, 'gpu_compute': 336.5687828063965, 'gpu_total': 336.9982543945313} 2022-11-23T03:11:33.4616944Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4617826Z warnings.warn( 2022-11-23T03:11:33.4618066Z ok (24.563s) 2022-11-23T03:11:33.4618624Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 9 2022-11-23T03:11:33.4619156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45159 2022-11-23T03:11:33.4619811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4620281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4620903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4621392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4621862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4622612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4623170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4623533Z dist init r=0, world=1 2022-11-23T03:11:33.4623691Z 2022-11-23T03:11:33.4623776Z rank0: 2022-11-23T03:11:33.4624301Z e1: {'cpu_iter': 0.002928604100000065, 'cpu_wait': 4.077000000002329e-05, 'gpu_compute': 0.076431999867782, 'gpu_total': 0.7082890033721924} 2022-11-23T03:11:33.4624943Z e2: {'cpu_iter': 0.010589585499999998, 'cpu_wait': 3.142199999999207e-05, 'gpu_compute': 0.21665600016713144, 'gpu_total': 2.141507160663605} 2022-11-23T03:11:33.4625567Z e3: {'cpu_iter': 0.004992504699999856, 'cpu_wait': 0.581128616, 'gpu_compute': 336.2116214752197, 'gpu_total': 336.43360595703126} 2022-11-23T03:11:33.4626187Z e4: {'cpu_iter': 0.015169318799999942, 'cpu_wait': 0.5753117282000005, 'gpu_compute': 335.5442617416382, 'gpu_total': 335.97491149902345} 2022-11-23T03:11:33.4627577Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4628407Z warnings.warn( 2022-11-23T03:11:33.4628631Z ok (58.740s) 2022-11-23T03:11:33.4629202Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 8 2022-11-23T03:11:33.4629749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45225 2022-11-23T03:11:33.4630414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4630892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4631516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4632007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4632466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4633180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4633734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4634150Z dist init r=0, world=1 2022-11-23T03:11:33.4638906Z 2022-11-23T03:11:33.4639047Z rank0: 2022-11-23T03:11:33.4639651Z e1: {'cpu_iter': 0.0028811012999997664, 'cpu_wait': 4.497140000001565e-05, 'gpu_compute': 0.0694882005918771, 'gpu_total': 0.6583851993083953} 2022-11-23T03:11:33.4640376Z e2: {'cpu_iter': 0.01056053659999998, 'cpu_wait': 3.1717400000097486e-05, 'gpu_compute': 0.22360059916973113, 'gpu_total': 2.197539234161377} 2022-11-23T03:11:33.4641016Z e3: {'cpu_iter': 0.0048174024000003255, 'cpu_wait': 0.5860746216000001, 'gpu_compute': 338.8785087585449, 'gpu_total': 339.10535888671876} 2022-11-23T03:11:33.4641614Z e4: {'cpu_iter': 0.0157584748999998, 'cpu_wait': 0.5797382634999998, 'gpu_compute': 337.97562103271486, 'gpu_total': 338.4027893066406} 2022-11-23T03:11:33.4643010Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4643855Z warnings.warn( 2022-11-23T03:11:33.4644097Z ok (19.456s) 2022-11-23T03:11:33.4644790Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 7 2022-11-23T03:11:33.4645339Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45291 2022-11-23T03:11:33.4646007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4646478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4647108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4647589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4648059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4648783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4649343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4649713Z dist init r=0, world=1 2022-11-23T03:11:33.4649874Z 2022-11-23T03:11:33.4649958Z rank0: 2022-11-23T03:11:33.4650479Z e1: {'cpu_iter': 0.0028744382999998487, 'cpu_wait': 3.9935799999923914e-05, 'gpu_compute': 0.0696319991722703, 'gpu_total': 0.6213605999946594} 2022-11-23T03:11:33.4651113Z e2: {'cpu_iter': 0.010564004399999938, 'cpu_wait': 3.453990000004126e-05, 'gpu_compute': 0.20267260018736125, 'gpu_total': 2.1003291964530946} 2022-11-23T03:11:33.4651730Z e3: {'cpu_iter': 0.005220080200000155, 'cpu_wait': 0.5879421183000009, 'gpu_compute': 339.7656425476074, 'gpu_total': 340.0173858642578} 2022-11-23T03:11:33.4652353Z e4: {'cpu_iter': 0.01580195169999854, 'cpu_wait': 0.5861195615000007, 'gpu_compute': 341.1389778137207, 'gpu_total': 341.598486328125} 2022-11-23T03:11:33.4653736Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4654574Z warnings.warn( 2022-11-23T03:11:33.4654817Z ok (25.671s) 2022-11-23T03:11:33.4655390Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 6 2022-11-23T03:11:33.4656023Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45357 2022-11-23T03:11:33.4656680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4657134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4657765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4658256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4658732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4659451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4660008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4660375Z dist init r=0, world=1 2022-11-23T03:11:33.4660541Z 2022-11-23T03:11:33.4660624Z rank0: 2022-11-23T03:11:33.4661135Z e1: {'cpu_iter': 0.0028839772999999626, 'cpu_wait': 4.338139999999768e-05, 'gpu_compute': 0.07721620001830161, 'gpu_total': 0.7077283978462219} 2022-11-23T03:11:33.4661820Z e2: {'cpu_iter': 0.01057262649999995, 'cpu_wait': 3.173309999970897e-05, 'gpu_compute': 0.19484819965437056, 'gpu_total': 2.0692675948143004} 2022-11-23T03:11:33.4662450Z e3: {'cpu_iter': 0.005103486400000001, 'cpu_wait': 0.5751694538999995, 'gpu_compute': 331.6012790679932, 'gpu_total': 331.8213653564453} 2022-11-23T03:11:33.4663068Z e4: {'cpu_iter': 0.016171100900000467, 'cpu_wait': 0.5783600460999996, 'gpu_compute': 337.2857955932617, 'gpu_total': 337.71690979003904} 2022-11-23T03:11:33.4664451Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4665278Z warnings.warn( 2022-11-23T03:11:33.4665515Z ok (17.751s) 2022-11-23T03:11:33.4666092Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 5 2022-11-23T03:11:33.4666635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45423 2022-11-23T03:11:33.4667279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4667738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4668358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4668848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4669328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4670050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4670597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4670956Z dist init r=0, world=1 2022-11-23T03:11:33.4671111Z 2022-11-23T03:11:33.4671195Z rank0: 2022-11-23T03:11:33.4671726Z e1: {'cpu_iter': 0.0028486460000001655, 'cpu_wait': 4.501409999990713e-05, 'gpu_compute': 0.08088000020943582, 'gpu_total': 0.7045821964740753} 2022-11-23T03:11:33.4672358Z e2: {'cpu_iter': 0.01075971399999993, 'cpu_wait': 3.220839999995562e-05, 'gpu_compute': 0.20876839980483056, 'gpu_total': 2.1003855586051943} 2022-11-23T03:11:33.4672977Z e3: {'cpu_iter': 0.005520187999999937, 'cpu_wait': 0.5761588234000001, 'gpu_compute': 334.23623580932616, 'gpu_total': 334.4747772216797} 2022-11-23T03:11:33.4673644Z e4: {'cpu_iter': 0.015754529500000204, 'cpu_wait': 0.5660711798000004, 'gpu_compute': 331.07121562957764, 'gpu_total': 331.53224487304686} 2022-11-23T03:11:33.4675032Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4675865Z warnings.warn( 2022-11-23T03:11:33.4676112Z ok (17.655s) 2022-11-23T03:11:33.4676680Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 4 2022-11-23T03:11:33.4677200Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45489 2022-11-23T03:11:33.4677855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4678531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4679203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4679695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4680224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4680946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4681485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4681849Z dist init r=0, world=1 2022-11-23T03:11:33.4682009Z 2022-11-23T03:11:33.4682096Z rank0: 2022-11-23T03:11:33.4682625Z e1: {'cpu_iter': 0.003035560699999795, 'cpu_wait': 4.308040000005065e-05, 'gpu_compute': 0.08071999987587333, 'gpu_total': 0.7059529960155487} 2022-11-23T03:11:33.4683265Z e2: {'cpu_iter': 0.010506064199999976, 'cpu_wait': 3.206260000006011e-05, 'gpu_compute': 0.20278419964015484, 'gpu_total': 2.0838382124900816} 2022-11-23T03:11:33.4683900Z e3: {'cpu_iter': 0.0051524698999998895, 'cpu_wait': 0.5902245761999999, 'gpu_compute': 340.54668045043945, 'gpu_total': 340.76930236816406} 2022-11-23T03:11:33.4684521Z e4: {'cpu_iter': 0.015814184599999236, 'cpu_wait': 0.5866695595000003, 'gpu_compute': 340.0502502441406, 'gpu_total': 340.4747650146484} 2022-11-23T03:11:33.4685913Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4686737Z warnings.warn( 2022-11-23T03:11:33.4686964Z ok (24.166s) 2022-11-23T03:11:33.4687543Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 3 2022-11-23T03:11:33.4688080Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45555 2022-11-23T03:11:33.4688735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4689203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4689829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4690322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4690857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4691562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4692111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4692476Z dist init r=0, world=1 2022-11-23T03:11:33.4692635Z 2022-11-23T03:11:33.4692723Z rank0: 2022-11-23T03:11:33.4693254Z e1: {'cpu_iter': 0.002796066699999944, 'cpu_wait': 4.286389999998974e-05, 'gpu_compute': 0.07216000016778708, 'gpu_total': 0.6457922041416169} 2022-11-23T03:11:33.4693891Z e2: {'cpu_iter': 0.01053274309999983, 'cpu_wait': 3.168660000003598e-05, 'gpu_compute': 0.1998573996126652, 'gpu_total': 2.061003398895264} 2022-11-23T03:11:33.4694514Z e3: {'cpu_iter': 0.004193346599999881, 'cpu_wait': 0.5801242381000004, 'gpu_compute': 335.3048713684082, 'gpu_total': 335.5352416992188} 2022-11-23T03:11:33.4695128Z e4: {'cpu_iter': 0.014016573699999313, 'cpu_wait': 0.5867003726000014, 'gpu_compute': 340.8659149169922, 'gpu_total': 341.2935394287109} 2022-11-23T03:11:33.4696544Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4697371Z warnings.warn( 2022-11-23T03:11:33.4697598Z ok (21.962s) 2022-11-23T03:11:33.4698169Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 2 2022-11-23T03:11:33.4698702Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45621 2022-11-23T03:11:33.4699360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4699826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4700451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4700926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4701406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4702118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4702667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4703030Z dist init r=0, world=1 2022-11-23T03:11:33.4703190Z 2022-11-23T03:11:33.4703275Z rank0: 2022-11-23T03:11:33.4703799Z e1: {'cpu_iter': 0.002918400899999796, 'cpu_wait': 4.2919399999874484e-05, 'gpu_compute': 0.07419219994917511, 'gpu_total': 0.6758248031139373} 2022-11-23T03:11:33.4704435Z e2: {'cpu_iter': 0.010538775300000048, 'cpu_wait': 3.117839999990935e-05, 'gpu_compute': 0.21195199936628342, 'gpu_total': 2.1241379976272583} 2022-11-23T03:11:33.4705053Z e3: {'cpu_iter': 0.0053495242000000335, 'cpu_wait': 0.5626030393000001, 'gpu_compute': 327.17699203491213, 'gpu_total': 327.4031921386719} 2022-11-23T03:11:33.4705675Z e4: {'cpu_iter': 0.015915940699999707, 'cpu_wait': 0.5859206106999999, 'gpu_compute': 341.03024368286134, 'gpu_total': 341.4576354980469} 2022-11-23T03:11:33.4707063Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4707930Z warnings.warn( 2022-11-23T03:11:33.4708176Z ok (30.582s) 2022-11-23T03:11:33.4708755Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... test_forward_overlap succeeded - num_retries_left: 1 2022-11-23T03:11:33.4709304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45687 2022-11-23T03:11:33.4709965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4710436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4711046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4711541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4712020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4712744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:11:33.4713300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4713723Z dist init r=0, world=1 2022-11-23T03:11:33.4713884Z 2022-11-23T03:11:33.4713974Z rank0: 2022-11-23T03:11:33.4714495Z e1: {'cpu_iter': 0.0027203948999998673, 'cpu_wait': 4.3375800000067247e-05, 'gpu_compute': 0.06891240044496953, 'gpu_total': 0.6360483944416047} 2022-11-23T03:11:33.4715135Z e2: {'cpu_iter': 0.010623461800000022, 'cpu_wait': 3.210259999990584e-05, 'gpu_compute': 0.22536019794642925, 'gpu_total': 2.215487778186798} 2022-11-23T03:11:33.4715761Z e3: {'cpu_iter': 0.005380171999999827, 'cpu_wait': 0.5733180046000002, 'gpu_compute': 332.6690299987793, 'gpu_total': 332.90296325683596} 2022-11-23T03:11:33.4716378Z e4: {'cpu_iter': 0.015305403200000355, 'cpu_wait': 0.5754559895999993, 'gpu_compute': 335.5659812927246, 'gpu_total': 336.0122863769531} 2022-11-23T03:11:33.4717768Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4718590Z warnings.warn( 2022-11-23T03:11:33.4718840Z ok (23.466s) 2022-11-23T03:11:33.4719243Z test_forward_overlap succeeded - num_retries_left: 0 2022-11-23T03:11:33.4719795Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45753 2022-11-23T03:11:33.4720429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45754 2022-11-23T03:11:33.4721077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4721539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4722163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4722651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4723128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4723812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4724282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4724894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4725438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4725907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4726620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4727368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4727925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4728424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4729845Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4730682Z warnings.warn( 2022-11-23T03:11:33.4731078Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4731440Z Traceback (most recent call last): 2022-11-23T03:11:33.4732025Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4732439Z getattr(self, test_name)() 2022-11-23T03:11:33.4732999Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4733383Z fn() 2022-11-23T03:11:33.4733895Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4734308Z return func(*args, **kwargs) 2022-11-23T03:11:33.4734726Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4735106Z self._dist_train() 2022-11-23T03:11:33.4735498Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4736010Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4736438Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4736817Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4737191Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4737561Z raise self.failureException(msg) 2022-11-23T03:11:33.4737878Z AssertionError: False is not true 2022-11-23T03:11:33.4738187Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4738458Z dist init r=0, world=2 2022-11-23T03:11:33.4739769Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4740602Z warnings.warn( 2022-11-23T03:11:33.4740943Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4741310Z Traceback (most recent call last): 2022-11-23T03:11:33.4741886Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4742293Z getattr(self, test_name)() 2022-11-23T03:11:33.4742848Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4743268Z fn() 2022-11-23T03:11:33.4743782Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4744176Z return func(*args, **kwargs) 2022-11-23T03:11:33.4744595Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4744973Z self._dist_train() 2022-11-23T03:11:33.4745361Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4745880Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4746305Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4746676Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4747046Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4747408Z raise self.failureException(msg) 2022-11-23T03:11:33.4747728Z AssertionError: False is not true 2022-11-23T03:11:33.4748031Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4748296Z dist init r=1, world=2 2022-11-23T03:11:33.4748649Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4748980Z ERROR (3.733s) 2022-11-23T03:11:33.4749606Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 50 2022-11-23T03:11:33.4750057Z Traceback (most recent call last): 2022-11-23T03:11:33.4750630Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4751037Z self._join_processes(fn) 2022-11-23T03:11:33.4751600Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4752042Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4752645Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4753075Z raise RuntimeError(error) 2022-11-23T03:11:33.4753426Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.4753771Z Traceback (most recent call last): 2022-11-23T03:11:33.4754333Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4754733Z getattr(self, test_name)() 2022-11-23T03:11:33.4755289Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4755664Z fn() 2022-11-23T03:11:33.4756178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4756581Z return func(*args, **kwargs) 2022-11-23T03:11:33.4756991Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4757360Z self._dist_train() 2022-11-23T03:11:33.4757744Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4758269Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4758701Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4759090Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4759460Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4759816Z raise self.failureException(msg) 2022-11-23T03:11:33.4760193Z AssertionError: False is not true 2022-11-23T03:11:33.4760375Z 2022-11-23T03:11:33.4760381Z 2022-11-23T03:11:33.4760386Z 2022-11-23T03:11:33.4760611Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45884 2022-11-23T03:11:33.4761089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45885 2022-11-23T03:11:33.4761816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4762287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4762912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4763399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4763856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4764532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4764999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4765613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4766097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4766566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4767330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4768084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4768615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4769110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4770512Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4770625Z warnings.warn( 2022-11-23T03:11:33.4770832Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4770962Z Traceback (most recent call last): 2022-11-23T03:11:33.4771362Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4771478Z getattr(self, test_name)() 2022-11-23T03:11:33.4771876Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4771964Z fn() 2022-11-23T03:11:33.4772351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4772472Z return func(*args, **kwargs) 2022-11-23T03:11:33.4772720Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4772822Z self._dist_train() 2022-11-23T03:11:33.4773058Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4773314Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4773524Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4773657Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4773850Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4773988Z raise self.failureException(msg) 2022-11-23T03:11:33.4774118Z AssertionError: False is not true 2022-11-23T03:11:33.4774250Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4774352Z dist init r=1, world=2 2022-11-23T03:11:33.4775555Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4775660Z warnings.warn( 2022-11-23T03:11:33.4775866Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4775994Z Traceback (most recent call last): 2022-11-23T03:11:33.4776388Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4776504Z getattr(self, test_name)() 2022-11-23T03:11:33.4776893Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4776985Z fn() 2022-11-23T03:11:33.4777366Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4777484Z return func(*args, **kwargs) 2022-11-23T03:11:33.4777807Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4777916Z self._dist_train() 2022-11-23T03:11:33.4778139Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4778394Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4778624Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4778920Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4779116Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4779259Z raise self.failureException(msg) 2022-11-23T03:11:33.4779396Z AssertionError: False is not true 2022-11-23T03:11:33.4779535Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4779645Z dist init r=0, world=2 2022-11-23T03:11:33.4779853Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4779956Z ERROR (3.531s) 2022-11-23T03:11:33.4780401Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 49 2022-11-23T03:11:33.4780534Z Traceback (most recent call last): 2022-11-23T03:11:33.4780929Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4781050Z self._join_processes(fn) 2022-11-23T03:11:33.4781461Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4781590Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4782014Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4782143Z raise RuntimeError(error) 2022-11-23T03:11:33.4782336Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.4782470Z Traceback (most recent call last): 2022-11-23T03:11:33.4782872Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4782992Z getattr(self, test_name)() 2022-11-23T03:11:33.4783388Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4783482Z fn() 2022-11-23T03:11:33.4783872Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4783994Z return func(*args, **kwargs) 2022-11-23T03:11:33.4784249Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4784408Z self._dist_train() 2022-11-23T03:11:33.4784650Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4784913Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4785147Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4785286Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4785464Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4785610Z raise self.failureException(msg) 2022-11-23T03:11:33.4785742Z AssertionError: False is not true 2022-11-23T03:11:33.4785750Z 2022-11-23T03:11:33.4785756Z 2022-11-23T03:11:33.4785763Z 2022-11-23T03:11:33.4785989Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46015 2022-11-23T03:11:33.4786212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46016 2022-11-23T03:11:33.4786623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4786804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4787260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4787459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4787716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4788121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4788298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4788710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4788909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4789166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4789606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4790044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4790283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4790509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4791659Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4791776Z warnings.warn( 2022-11-23T03:11:33.4791990Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4792127Z Traceback (most recent call last): 2022-11-23T03:11:33.4792530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4792654Z getattr(self, test_name)() 2022-11-23T03:11:33.4793040Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4793136Z fn() 2022-11-23T03:11:33.4793533Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4793657Z return func(*args, **kwargs) 2022-11-23T03:11:33.4793991Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4794100Z self._dist_train() 2022-11-23T03:11:33.4794343Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4794606Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4794836Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4794972Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4795172Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4795318Z raise self.failureException(msg) 2022-11-23T03:11:33.4795450Z AssertionError: False is not true 2022-11-23T03:11:33.4795585Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4795693Z dist init r=1, world=2 2022-11-23T03:11:33.4796886Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4797000Z warnings.warn( 2022-11-23T03:11:33.4797209Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4797343Z Traceback (most recent call last): 2022-11-23T03:11:33.4797727Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4797850Z getattr(self, test_name)() 2022-11-23T03:11:33.4798246Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4798345Z fn() 2022-11-23T03:11:33.4798736Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4798859Z return func(*args, **kwargs) 2022-11-23T03:11:33.4799114Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4799223Z self._dist_train() 2022-11-23T03:11:33.4799464Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4799722Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4799952Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4800091Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4800365Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4800511Z raise self.failureException(msg) 2022-11-23T03:11:33.4800650Z AssertionError: False is not true 2022-11-23T03:11:33.4800788Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4800878Z dist init r=0, world=2 2022-11-23T03:11:33.4801086Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4801189Z ERROR (3.733s) 2022-11-23T03:11:33.4801632Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 48 2022-11-23T03:11:33.4801768Z Traceback (most recent call last): 2022-11-23T03:11:33.4802167Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4802357Z self._join_processes(fn) 2022-11-23T03:11:33.4802773Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4802918Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4803338Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4803526Z raise RuntimeError(error) 2022-11-23T03:11:33.4803718Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.4803851Z Traceback (most recent call last): 2022-11-23T03:11:33.4804255Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4804372Z getattr(self, test_name)() 2022-11-23T03:11:33.4804763Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4804856Z fn() 2022-11-23T03:11:33.4805232Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4805352Z return func(*args, **kwargs) 2022-11-23T03:11:33.4805608Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4805717Z self._dist_train() 2022-11-23T03:11:33.4805959Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4806219Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4806499Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4806639Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4806837Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4806978Z raise self.failureException(msg) 2022-11-23T03:11:33.4807111Z AssertionError: False is not true 2022-11-23T03:11:33.4807119Z 2022-11-23T03:11:33.4807126Z 2022-11-23T03:11:33.4807132Z 2022-11-23T03:11:33.4807356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46146 2022-11-23T03:11:33.4807581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46147 2022-11-23T03:11:33.4807989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4808167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4808581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4808776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4809031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4809434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4809615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4810031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4810212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4810467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4810903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4811345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4811581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4811815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4812962Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4813117Z warnings.warn( 2022-11-23T03:11:33.4813327Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4813466Z Traceback (most recent call last): 2022-11-23T03:11:33.4813872Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4813992Z getattr(self, test_name)() 2022-11-23T03:11:33.4814391Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4814484Z fn() 2022-11-23T03:11:33.4814877Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4814999Z return func(*args, **kwargs) 2022-11-23T03:11:33.4815255Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4842040Z self._dist_train() 2022-11-23T03:11:33.4842382Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4842921Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4843162Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4843303Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4843485Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4843631Z raise self.failureException(msg) 2022-11-23T03:11:33.4843762Z AssertionError: False is not true 2022-11-23T03:11:33.4843894Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4844008Z dist init r=1, world=2 2022-11-23T03:11:33.4845187Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4845308Z warnings.warn( 2022-11-23T03:11:33.4845519Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4845655Z Traceback (most recent call last): 2022-11-23T03:11:33.4846061Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4846178Z getattr(self, test_name)() 2022-11-23T03:11:33.4846568Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4846654Z fn() 2022-11-23T03:11:33.4847046Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4847160Z return func(*args, **kwargs) 2022-11-23T03:11:33.4847411Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4847510Z self._dist_train() 2022-11-23T03:11:33.4847744Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4847992Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4848204Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4848332Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4848524Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4848665Z raise self.failureException(msg) 2022-11-23T03:11:33.4848795Z AssertionError: False is not true 2022-11-23T03:11:33.4848998Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4849100Z dist init r=0, world=2 2022-11-23T03:11:33.4849305Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4849400Z ERROR (3.631s) 2022-11-23T03:11:33.4849837Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 47 2022-11-23T03:11:33.4849965Z Traceback (most recent call last): 2022-11-23T03:11:33.4850353Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4850472Z self._join_processes(fn) 2022-11-23T03:11:33.4850883Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4851024Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4851441Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4851565Z raise RuntimeError(error) 2022-11-23T03:11:33.4851740Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.4851867Z Traceback (most recent call last): 2022-11-23T03:11:33.4852304Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4852423Z getattr(self, test_name)() 2022-11-23T03:11:33.4852811Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4852893Z fn() 2022-11-23T03:11:33.4853272Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4853385Z return func(*args, **kwargs) 2022-11-23T03:11:33.4853632Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4853735Z self._dist_train() 2022-11-23T03:11:33.4853969Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4854221Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4854445Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4854575Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4854764Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4854900Z raise self.failureException(msg) 2022-11-23T03:11:33.4855016Z AssertionError: False is not true 2022-11-23T03:11:33.4855033Z 2022-11-23T03:11:33.4855040Z 2022-11-23T03:11:33.4855046Z 2022-11-23T03:11:33.4855265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46277 2022-11-23T03:11:33.4855471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46278 2022-11-23T03:11:33.4855875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4856046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4856452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4856643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4856892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4857282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4857453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4857858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4858043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4858340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4858772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4859205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4859437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4859661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4860801Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4860905Z warnings.warn( 2022-11-23T03:11:33.4861107Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4861278Z Traceback (most recent call last): 2022-11-23T03:11:33.4861676Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4861789Z getattr(self, test_name)() 2022-11-23T03:11:33.4862175Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4862257Z fn() 2022-11-23T03:11:33.4862638Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4862742Z return func(*args, **kwargs) 2022-11-23T03:11:33.4862993Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4863091Z self._dist_train() 2022-11-23T03:11:33.4863324Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4863575Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4863796Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4863924Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4864110Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4864243Z raise self.failureException(msg) 2022-11-23T03:11:33.4864366Z AssertionError: False is not true 2022-11-23T03:11:33.4864491Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4864586Z dist init r=1, world=2 2022-11-23T03:11:33.4865725Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4865825Z warnings.warn( 2022-11-23T03:11:33.4866021Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4866142Z Traceback (most recent call last): 2022-11-23T03:11:33.4866531Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4866640Z getattr(self, test_name)() 2022-11-23T03:11:33.4867022Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4867097Z fn() 2022-11-23T03:11:33.4867532Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4867643Z return func(*args, **kwargs) 2022-11-23T03:11:33.4867886Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4867984Z self._dist_train() 2022-11-23T03:11:33.4868219Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4868467Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4868687Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4868815Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4868999Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4869132Z raise self.failureException(msg) 2022-11-23T03:11:33.4869255Z AssertionError: False is not true 2022-11-23T03:11:33.4869380Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4869478Z dist init r=0, world=2 2022-11-23T03:11:33.4869678Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4869769Z ERROR (3.932s) 2022-11-23T03:11:33.4870235Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 46 2022-11-23T03:11:33.4870359Z Traceback (most recent call last): 2022-11-23T03:11:33.4870746Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4870855Z self._join_processes(fn) 2022-11-23T03:11:33.4871257Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4871394Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4871804Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4871923Z raise RuntimeError(error) 2022-11-23T03:11:33.4872103Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.4872224Z Traceback (most recent call last): 2022-11-23T03:11:33.4872609Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4872719Z getattr(self, test_name)() 2022-11-23T03:11:33.4873098Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4873180Z fn() 2022-11-23T03:11:33.4873561Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4873671Z return func(*args, **kwargs) 2022-11-23T03:11:33.4873918Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4874010Z self._dist_train() 2022-11-23T03:11:33.4874238Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4874484Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4874702Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4874834Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4875017Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4875152Z raise self.failureException(msg) 2022-11-23T03:11:33.4875277Z AssertionError: False is not true 2022-11-23T03:11:33.4875284Z 2022-11-23T03:11:33.4875290Z 2022-11-23T03:11:33.4875296Z 2022-11-23T03:11:33.4875514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46408 2022-11-23T03:11:33.4875726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46409 2022-11-23T03:11:33.4876121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4876361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4876764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4876951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4877193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4877585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4877750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4878158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4878339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4878586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4879014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4879499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4879722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4879942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4881152Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4881261Z warnings.warn( 2022-11-23T03:11:33.4881453Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4881578Z Traceback (most recent call last): 2022-11-23T03:11:33.4881973Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4882083Z getattr(self, test_name)() 2022-11-23T03:11:33.4882470Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4882554Z fn() 2022-11-23T03:11:33.4882934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4883046Z return func(*args, **kwargs) 2022-11-23T03:11:33.4883294Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4883392Z self._dist_train() 2022-11-23T03:11:33.4883625Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4883869Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4884090Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4884217Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4884401Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4884531Z raise self.failureException(msg) 2022-11-23T03:11:33.4884647Z AssertionError: False is not true 2022-11-23T03:11:33.4884772Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4884867Z dist init r=1, world=2 2022-11-23T03:11:33.4886080Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4886236Z warnings.warn( 2022-11-23T03:11:33.4886432Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4886557Z Traceback (most recent call last): 2022-11-23T03:11:33.4886957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4887076Z getattr(self, test_name)() 2022-11-23T03:11:33.4887467Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4887564Z fn() 2022-11-23T03:11:33.4887951Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4888073Z return func(*args, **kwargs) 2022-11-23T03:11:33.4888324Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4888430Z self._dist_train() 2022-11-23T03:11:33.4888854Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4889113Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4889344Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4889476Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4889657Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4889796Z raise self.failureException(msg) 2022-11-23T03:11:33.4889928Z AssertionError: False is not true 2022-11-23T03:11:33.4890062Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4890167Z dist init r=0, world=2 2022-11-23T03:11:33.4890367Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4890462Z ERROR (3.631s) 2022-11-23T03:11:33.4890898Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 45 2022-11-23T03:11:33.4891028Z Traceback (most recent call last): 2022-11-23T03:11:33.4891423Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4891535Z self._join_processes(fn) 2022-11-23T03:11:33.4891948Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4892090Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4892503Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4892629Z raise RuntimeError(error) 2022-11-23T03:11:33.4892816Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.4892932Z Traceback (most recent call last): 2022-11-23T03:11:33.4893330Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4893446Z getattr(self, test_name)() 2022-11-23T03:11:33.4893836Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4893929Z fn() 2022-11-23T03:11:33.4894315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4894436Z return func(*args, **kwargs) 2022-11-23T03:11:33.4894689Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4894791Z self._dist_train() 2022-11-23T03:11:33.4895079Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4895334Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4895560Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4895695Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4895887Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4896027Z raise self.failureException(msg) 2022-11-23T03:11:33.4896158Z AssertionError: False is not true 2022-11-23T03:11:33.4896166Z 2022-11-23T03:11:33.4896173Z 2022-11-23T03:11:33.4896179Z 2022-11-23T03:11:33.4896401Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46539 2022-11-23T03:11:33.4896621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46540 2022-11-23T03:11:33.4897023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4897199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4897600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4897837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4898091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4898492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4898668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4899076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4899272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4899527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4899969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4900402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4900637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4900867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4902015Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4902125Z warnings.warn( 2022-11-23T03:11:33.4902328Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4902460Z Traceback (most recent call last): 2022-11-23T03:11:33.4902859Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4902979Z getattr(self, test_name)() 2022-11-23T03:11:33.4903373Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4903467Z fn() 2022-11-23T03:11:33.4903852Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4903968Z return func(*args, **kwargs) 2022-11-23T03:11:33.4904222Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4904360Z self._dist_train() 2022-11-23T03:11:33.4904599Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4904855Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4905087Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4905224Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4905416Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4905555Z raise self.failureException(msg) 2022-11-23T03:11:33.4905687Z AssertionError: False is not true 2022-11-23T03:11:33.4905819Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4905923Z dist init r=0, world=2 2022-11-23T03:11:33.4907069Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4907228Z warnings.warn( 2022-11-23T03:11:33.4907436Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4907563Z Traceback (most recent call last): 2022-11-23T03:11:33.4907963Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4908079Z getattr(self, test_name)() 2022-11-23T03:11:33.4908471Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4908556Z fn() 2022-11-23T03:11:33.4908934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4909043Z return func(*args, **kwargs) 2022-11-23T03:11:33.4909296Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4909398Z self._dist_train() 2022-11-23T03:11:33.4909637Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4909893Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4910119Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4910253Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4910444Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4910582Z raise self.failureException(msg) 2022-11-23T03:11:33.4910710Z AssertionError: False is not true 2022-11-23T03:11:33.4910841Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4910944Z dist init r=1, world=2 2022-11-23T03:11:33.4911146Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4911242Z ERROR (3.631s) 2022-11-23T03:11:33.4911672Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 44 2022-11-23T03:11:33.4911805Z Traceback (most recent call last): 2022-11-23T03:11:33.4912186Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4912297Z self._join_processes(fn) 2022-11-23T03:11:33.4912709Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4912854Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4913266Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4913437Z raise RuntimeError(error) 2022-11-23T03:11:33.4913628Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.4913759Z Traceback (most recent call last): 2022-11-23T03:11:33.4914152Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4914271Z getattr(self, test_name)() 2022-11-23T03:11:33.4914661Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4914747Z fn() 2022-11-23T03:11:33.4915132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4915250Z return func(*args, **kwargs) 2022-11-23T03:11:33.4915499Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4915602Z self._dist_train() 2022-11-23T03:11:33.4915844Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4916083Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4916309Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4916491Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4916690Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4916831Z raise self.failureException(msg) 2022-11-23T03:11:33.4916962Z AssertionError: False is not true 2022-11-23T03:11:33.4916970Z 2022-11-23T03:11:33.4916976Z 2022-11-23T03:11:33.4916982Z 2022-11-23T03:11:33.4917205Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46670 2022-11-23T03:11:33.4917427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46671 2022-11-23T03:11:33.4917835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4918013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4918422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4918620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4918873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4919272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4919447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4919859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4920051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4920366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4920804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4921234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4921471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4921706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4922849Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4923019Z warnings.warn( 2022-11-23T03:11:33.4923224Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4923340Z Traceback (most recent call last): 2022-11-23T03:11:33.4923740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4923858Z getattr(self, test_name)() 2022-11-23T03:11:33.4924251Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4924338Z fn() 2022-11-23T03:11:33.4924729Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4924852Z return func(*args, **kwargs) 2022-11-23T03:11:33.4925296Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4925408Z self._dist_train() 2022-11-23T03:11:33.4925646Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4925904Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4926184Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4926317Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4926509Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4926646Z raise self.failureException(msg) 2022-11-23T03:11:33.4926776Z AssertionError: False is not true 2022-11-23T03:11:33.4926906Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4928046Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4928156Z warnings.warn( 2022-11-23T03:11:33.4928361Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4928477Z Traceback (most recent call last): 2022-11-23T03:11:33.4928868Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4928986Z getattr(self, test_name)() 2022-11-23T03:11:33.4929378Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4929472Z fn() 2022-11-23T03:11:33.4929859Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4929983Z return func(*args, **kwargs) 2022-11-23T03:11:33.4930237Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4930343Z self._dist_train() 2022-11-23T03:11:33.4930579Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4930831Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4931052Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4931189Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4931384Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4931525Z raise self.failureException(msg) 2022-11-23T03:11:33.4931653Z AssertionError: False is not true 2022-11-23T03:11:33.4931770Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4931874Z dist init r=0, world=2 2022-11-23T03:11:33.4932021Z dist init r=1, world=2 2022-11-23T03:11:33.4932223Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4932322Z ERROR (3.632s) 2022-11-23T03:11:33.4932763Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 43 2022-11-23T03:11:33.4932898Z Traceback (most recent call last): 2022-11-23T03:11:33.4933290Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4933409Z self._join_processes(fn) 2022-11-23T03:11:33.4933825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4933966Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4934381Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4934507Z raise RuntimeError(error) 2022-11-23T03:11:33.4934695Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.4934826Z Traceback (most recent call last): 2022-11-23T03:11:33.4935216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4935377Z getattr(self, test_name)() 2022-11-23T03:11:33.4935759Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4935845Z fn() 2022-11-23T03:11:33.4936239Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4936359Z return func(*args, **kwargs) 2022-11-23T03:11:33.4936612Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4936718Z self._dist_train() 2022-11-23T03:11:33.4936959Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4937213Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4937439Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4937574Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4937768Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4937907Z raise self.failureException(msg) 2022-11-23T03:11:33.4938040Z AssertionError: False is not true 2022-11-23T03:11:33.4938048Z 2022-11-23T03:11:33.4938054Z 2022-11-23T03:11:33.4938059Z 2022-11-23T03:11:33.4938287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46801 2022-11-23T03:11:33.4938505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46802 2022-11-23T03:11:33.4938906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4939084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4939494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4939689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4939936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4940320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4940495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4940906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4941097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4941399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4941834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4942275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4942513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4942748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4943892Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4944009Z warnings.warn( 2022-11-23T03:11:33.4944212Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4944344Z Traceback (most recent call last): 2022-11-23T03:11:33.4944807Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4944921Z getattr(self, test_name)() 2022-11-23T03:11:33.4945322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4945412Z fn() 2022-11-23T03:11:33.4945798Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4945917Z return func(*args, **kwargs) 2022-11-23T03:11:33.4946170Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4946277Z self._dist_train() 2022-11-23T03:11:33.4946518Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4946760Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4946987Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4947123Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4947320Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4947457Z raise self.failureException(msg) 2022-11-23T03:11:33.4947587Z AssertionError: False is not true 2022-11-23T03:11:33.4947718Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4947826Z dist init r=1, world=2 2022-11-23T03:11:33.4948972Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4949080Z warnings.warn( 2022-11-23T03:11:33.4949289Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4949417Z Traceback (most recent call last): 2022-11-23T03:11:33.4949812Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4949932Z getattr(self, test_name)() 2022-11-23T03:11:33.4950322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4950412Z fn() 2022-11-23T03:11:33.4950798Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4950967Z return func(*args, **kwargs) 2022-11-23T03:11:33.4951220Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4951311Z self._dist_train() 2022-11-23T03:11:33.4951549Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4951805Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4952031Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4952165Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4952358Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4952498Z raise self.failureException(msg) 2022-11-23T03:11:33.4952628Z AssertionError: False is not true 2022-11-23T03:11:33.4952757Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4952865Z dist init r=0, world=2 2022-11-23T03:11:33.4953070Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4953170Z ERROR (3.531s) 2022-11-23T03:11:33.4953606Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 42 2022-11-23T03:11:33.4953785Z Traceback (most recent call last): 2022-11-23T03:11:33.4954184Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4954300Z self._join_processes(fn) 2022-11-23T03:11:33.4954697Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4954842Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4955257Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4955376Z raise RuntimeError(error) 2022-11-23T03:11:33.4955568Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.4955698Z Traceback (most recent call last): 2022-11-23T03:11:33.4956095Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4956222Z getattr(self, test_name)() 2022-11-23T03:11:33.4956614Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4956708Z fn() 2022-11-23T03:11:33.4957096Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4957212Z return func(*args, **kwargs) 2022-11-23T03:11:33.4957464Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4957567Z self._dist_train() 2022-11-23T03:11:33.4957808Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4958067Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4958296Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4958431Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4958612Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4958754Z raise self.failureException(msg) 2022-11-23T03:11:33.4958882Z AssertionError: False is not true 2022-11-23T03:11:33.4958889Z 2022-11-23T03:11:33.4958895Z 2022-11-23T03:11:33.4958901Z 2022-11-23T03:11:33.4959121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46932 2022-11-23T03:11:33.4959339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46933 2022-11-23T03:11:33.4959739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4959967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4960458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4960649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4960903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4961307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4961481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4961889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4962076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4962330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4962770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4963260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4963496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4963728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4964887Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4964999Z warnings.warn( 2022-11-23T03:11:33.4965204Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4965337Z Traceback (most recent call last): 2022-11-23T03:11:33.4965741Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4965860Z getattr(self, test_name)() 2022-11-23T03:11:33.4966240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4966329Z fn() 2022-11-23T03:11:33.4966716Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4966832Z return func(*args, **kwargs) 2022-11-23T03:11:33.4967091Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4967205Z self._dist_train() 2022-11-23T03:11:33.4967447Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4967699Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4967921Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4968057Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4968246Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4968387Z raise self.failureException(msg) 2022-11-23T03:11:33.4968516Z AssertionError: False is not true 2022-11-23T03:11:33.4968647Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4968752Z dist init r=1, world=2 2022-11-23T03:11:33.4969897Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4970055Z warnings.warn( 2022-11-23T03:11:33.4970268Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4970383Z Traceback (most recent call last): 2022-11-23T03:11:33.4970787Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4970904Z getattr(self, test_name)() 2022-11-23T03:11:33.4971299Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4971389Z fn() 2022-11-23T03:11:33.4971775Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4971897Z return func(*args, **kwargs) 2022-11-23T03:11:33.4972147Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4972251Z self._dist_train() 2022-11-23T03:11:33.4972536Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4972793Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4973014Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4973150Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4973342Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4973477Z raise self.failureException(msg) 2022-11-23T03:11:33.4973609Z AssertionError: False is not true 2022-11-23T03:11:33.4973740Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4973829Z dist init r=0, world=2 2022-11-23T03:11:33.4974039Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4974133Z ERROR (3.731s) 2022-11-23T03:11:33.4974568Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 41 2022-11-23T03:11:33.4974701Z Traceback (most recent call last): 2022-11-23T03:11:33.4975095Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4975213Z self._join_processes(fn) 2022-11-23T03:11:33.4975623Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4975766Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4976186Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4976310Z raise RuntimeError(error) 2022-11-23T03:11:33.4976500Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.4976626Z Traceback (most recent call last): 2022-11-23T03:11:33.4977018Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4977137Z getattr(self, test_name)() 2022-11-23T03:11:33.4977528Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4977616Z fn() 2022-11-23T03:11:33.4977988Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4978105Z return func(*args, **kwargs) 2022-11-23T03:11:33.4978359Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4978466Z self._dist_train() 2022-11-23T03:11:33.4978704Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4979010Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4979238Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4979374Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4979567Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4979699Z raise self.failureException(msg) 2022-11-23T03:11:33.4979828Z AssertionError: False is not true 2022-11-23T03:11:33.4979836Z 2022-11-23T03:11:33.4979841Z 2022-11-23T03:11:33.4979847Z 2022-11-23T03:11:33.4980071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47063 2022-11-23T03:11:33.4980293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47064 2022-11-23T03:11:33.4980700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4980878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4981291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4981483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4981783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.4982187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.4982362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.4982774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.4982950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.4983201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.4983639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4984073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.4984313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.4984543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.4985688Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4985800Z warnings.warn( 2022-11-23T03:11:33.4986012Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4986145Z Traceback (most recent call last): 2022-11-23T03:11:33.4986546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4986665Z getattr(self, test_name)() 2022-11-23T03:11:33.4987064Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4987153Z fn() 2022-11-23T03:11:33.4987542Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4987663Z return func(*args, **kwargs) 2022-11-23T03:11:33.4987916Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4988070Z self._dist_train() 2022-11-23T03:11:33.4988308Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4988563Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4988790Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4988922Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4989099Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4989239Z raise self.failureException(msg) 2022-11-23T03:11:33.4989369Z AssertionError: False is not true 2022-11-23T03:11:33.4989507Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.4990653Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.4990766Z warnings.warn( 2022-11-23T03:11:33.4990972Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.4991149Z Traceback (most recent call last): 2022-11-23T03:11:33.4991545Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4991658Z getattr(self, test_name)() 2022-11-23T03:11:33.4992046Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4992135Z fn() 2022-11-23T03:11:33.4992520Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4992637Z return func(*args, **kwargs) 2022-11-23T03:11:33.4992891Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4992994Z self._dist_train() 2022-11-23T03:11:33.4993230Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4993490Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.4993714Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.4993835Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.4994031Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.4994174Z raise self.failureException(msg) 2022-11-23T03:11:33.4994307Z AssertionError: False is not true 2022-11-23T03:11:33.4994441Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.4994550Z dist init r=0, world=2 2022-11-23T03:11:33.4994656Z dist init r=1, world=2 2022-11-23T03:11:33.4994861Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.4994960Z ERROR (3.731s) 2022-11-23T03:11:33.4995397Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 40 2022-11-23T03:11:33.4995530Z Traceback (most recent call last): 2022-11-23T03:11:33.4995925Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.4996041Z self._join_processes(fn) 2022-11-23T03:11:33.4996451Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.4996598Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.4997015Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.4997121Z raise RuntimeError(error) 2022-11-23T03:11:33.4997358Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.4997493Z Traceback (most recent call last): 2022-11-23T03:11:33.4997892Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.4998009Z getattr(self, test_name)() 2022-11-23T03:11:33.4998401Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.4998487Z fn() 2022-11-23T03:11:33.4998872Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.4998993Z return func(*args, **kwargs) 2022-11-23T03:11:33.4999246Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.4999352Z self._dist_train() 2022-11-23T03:11:33.4999591Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.4999849Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5000078Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5000269Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5000542Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5000685Z raise self.failureException(msg) 2022-11-23T03:11:33.5000801Z AssertionError: False is not true 2022-11-23T03:11:33.5000809Z 2022-11-23T03:11:33.5000832Z 2022-11-23T03:11:33.5000838Z 2022-11-23T03:11:33.5001046Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47194 2022-11-23T03:11:33.5001270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47195 2022-11-23T03:11:33.5001681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5001860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5002274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5002469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5002725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5003124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5003297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5003707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5003901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5004153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5004589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5005018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5005258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5005496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5006642Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5006812Z warnings.warn( 2022-11-23T03:11:33.5007017Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5007147Z Traceback (most recent call last): 2022-11-23T03:11:33.5007555Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5007677Z getattr(self, test_name)() 2022-11-23T03:11:33.5008079Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5008170Z fn() 2022-11-23T03:11:33.5008543Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5008661Z return func(*args, **kwargs) 2022-11-23T03:11:33.5008913Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5009020Z self._dist_train() 2022-11-23T03:11:33.5009260Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5009517Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5009749Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5009940Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5010136Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5010276Z raise self.failureException(msg) 2022-11-23T03:11:33.5010410Z AssertionError: False is not true 2022-11-23T03:11:33.5010543Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5011689Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5011803Z warnings.warn( 2022-11-23T03:11:33.5012008Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5012142Z Traceback (most recent call last): 2022-11-23T03:11:33.5012534Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5012650Z getattr(self, test_name)() 2022-11-23T03:11:33.5013043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5013120Z fn() 2022-11-23T03:11:33.5013503Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5013622Z return func(*args, **kwargs) 2022-11-23T03:11:33.5013877Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5013984Z self._dist_train() 2022-11-23T03:11:33.5014225Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5014481Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5014709Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5014848Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5015041Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5015184Z raise self.failureException(msg) 2022-11-23T03:11:33.5015317Z AssertionError: False is not true 2022-11-23T03:11:33.5015451Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5015559Z dist init r=1, world=2 2022-11-23T03:11:33.5015662Z dist init r=0, world=2 2022-11-23T03:11:33.5015867Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5016023Z ERROR (3.630s) 2022-11-23T03:11:33.5016466Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 39 2022-11-23T03:11:33.5016597Z Traceback (most recent call last): 2022-11-23T03:11:33.5016995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5017117Z self._join_processes(fn) 2022-11-23T03:11:33.5017530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5017681Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5018100Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5018227Z raise RuntimeError(error) 2022-11-23T03:11:33.5018413Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5018547Z Traceback (most recent call last): 2022-11-23T03:11:33.5018943Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5019063Z getattr(self, test_name)() 2022-11-23T03:11:33.5019500Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5019588Z fn() 2022-11-23T03:11:33.5019971Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5020092Z return func(*args, **kwargs) 2022-11-23T03:11:33.5020331Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5020434Z self._dist_train() 2022-11-23T03:11:33.5020669Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5020932Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5021160Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5021302Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5021501Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5021644Z raise self.failureException(msg) 2022-11-23T03:11:33.5021782Z AssertionError: False is not true 2022-11-23T03:11:33.5021789Z 2022-11-23T03:11:33.5021796Z 2022-11-23T03:11:33.5021802Z 2022-11-23T03:11:33.5022028Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47325 2022-11-23T03:11:33.5022250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47326 2022-11-23T03:11:33.5022659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5022835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5023248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5023440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5023687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5024090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5024266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5024679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5024871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5025123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5025611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5026046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5026288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5026519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5027669Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5027782Z warnings.warn( 2022-11-23T03:11:33.5027984Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5028100Z Traceback (most recent call last): 2022-11-23T03:11:33.5028542Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5028661Z getattr(self, test_name)() 2022-11-23T03:11:33.5029058Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5029147Z fn() 2022-11-23T03:11:33.5029539Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5029661Z return func(*args, **kwargs) 2022-11-23T03:11:33.5029914Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5030017Z self._dist_train() 2022-11-23T03:11:33.5030260Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5030513Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5030743Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5030883Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5031076Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5031216Z raise self.failureException(msg) 2022-11-23T03:11:33.5031344Z AssertionError: False is not true 2022-11-23T03:11:33.5031477Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5031566Z dist init r=1, world=2 2022-11-23T03:11:33.5032698Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5032818Z warnings.warn( 2022-11-23T03:11:33.5033007Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5033139Z Traceback (most recent call last): 2022-11-23T03:11:33.5033534Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5033655Z getattr(self, test_name)() 2022-11-23T03:11:33.5034047Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5034140Z fn() 2022-11-23T03:11:33.5034528Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5034649Z return func(*args, **kwargs) 2022-11-23T03:11:33.5034948Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5035053Z self._dist_train() 2022-11-23T03:11:33.5035293Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5035554Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5035780Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5035911Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5036103Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5036245Z raise self.failureException(msg) 2022-11-23T03:11:33.5036361Z AssertionError: False is not true 2022-11-23T03:11:33.5036493Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5036598Z dist init r=0, world=2 2022-11-23T03:11:33.5036799Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5036903Z ERROR (3.731s) 2022-11-23T03:11:33.5037344Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 38 2022-11-23T03:11:33.5037473Z Traceback (most recent call last): 2022-11-23T03:11:33.5037916Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5038039Z self._join_processes(fn) 2022-11-23T03:11:33.5038454Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5038600Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5039022Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5039308Z raise RuntimeError(error) 2022-11-23T03:11:33.5039503Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5039635Z Traceback (most recent call last): 2022-11-23T03:11:33.5040029Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5040211Z getattr(self, test_name)() 2022-11-23T03:11:33.5040591Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5040684Z fn() 2022-11-23T03:11:33.5041076Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5041198Z return func(*args, **kwargs) 2022-11-23T03:11:33.5041453Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5041560Z self._dist_train() 2022-11-23T03:11:33.5041798Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5042059Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5042292Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5042430Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5042625Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5042769Z raise self.failureException(msg) 2022-11-23T03:11:33.5042899Z AssertionError: False is not true 2022-11-23T03:11:33.5042908Z 2022-11-23T03:11:33.5042914Z 2022-11-23T03:11:33.5042920Z 2022-11-23T03:11:33.5043146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47456 2022-11-23T03:11:33.5043365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47457 2022-11-23T03:11:33.5043767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5043943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5044417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5044602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5044850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5045236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5045410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5045818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5046005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5046246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5046679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5047108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5047404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5047641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5048790Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5048901Z warnings.warn( 2022-11-23T03:11:33.5049106Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5049241Z Traceback (most recent call last): 2022-11-23T03:11:33.5049640Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5049764Z getattr(self, test_name)() 2022-11-23T03:11:33.5050160Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5050254Z fn() 2022-11-23T03:11:33.5050646Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5050770Z return func(*args, **kwargs) 2022-11-23T03:11:33.5051024Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5051133Z self._dist_train() 2022-11-23T03:11:33.5051379Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5051620Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5051849Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5051986Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5052182Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5052323Z raise self.failureException(msg) 2022-11-23T03:11:33.5052456Z AssertionError: False is not true 2022-11-23T03:11:33.5052588Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5053726Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5053883Z warnings.warn( 2022-11-23T03:11:33.5054084Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5054217Z Traceback (most recent call last): 2022-11-23T03:11:33.5054621Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5054744Z getattr(self, test_name)() 2022-11-23T03:11:33.5055140Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5055236Z fn() 2022-11-23T03:11:33.5055626Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5055747Z return func(*args, **kwargs) 2022-11-23T03:11:33.5056000Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5056107Z self._dist_train() 2022-11-23T03:11:33.5056328Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5056585Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5056856Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5056993Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5057188Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5057327Z raise self.failureException(msg) 2022-11-23T03:11:33.5057455Z AssertionError: False is not true 2022-11-23T03:11:33.5057587Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5057691Z dist init r=1, world=2 2022-11-23T03:11:33.5057797Z dist init r=0, world=2 2022-11-23T03:11:33.5058003Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5058107Z ERROR (3.731s) 2022-11-23T03:11:33.5058546Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 37 2022-11-23T03:11:33.5058677Z Traceback (most recent call last): 2022-11-23T03:11:33.5059072Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5059191Z self._join_processes(fn) 2022-11-23T03:11:33.5059587Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5059734Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5060158Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5060282Z raise RuntimeError(error) 2022-11-23T03:11:33.5060468Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5060600Z Traceback (most recent call last): 2022-11-23T03:11:33.5060993Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5061112Z getattr(self, test_name)() 2022-11-23T03:11:33.5061499Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5061591Z fn() 2022-11-23T03:11:33.5061976Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5062095Z return func(*args, **kwargs) 2022-11-23T03:11:33.5062347Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5062454Z self._dist_train() 2022-11-23T03:11:33.5062690Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5062944Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5063218Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5063356Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5063534Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5063678Z raise self.failureException(msg) 2022-11-23T03:11:33.5063805Z AssertionError: False is not true 2022-11-23T03:11:33.5063813Z 2022-11-23T03:11:33.5063819Z 2022-11-23T03:11:33.5063825Z 2022-11-23T03:11:33.5064052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47587 2022-11-23T03:11:33.5064271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47588 2022-11-23T03:11:33.5064683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5064856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5065273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5065470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5065775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5066185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5066361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5066770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5066963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5067213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5067648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5068080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5068316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5068550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5069696Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5069807Z warnings.warn( 2022-11-23T03:11:33.5070012Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5070146Z Traceback (most recent call last): 2022-11-23T03:11:33.5070539Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5070666Z getattr(self, test_name)() 2022-11-23T03:11:33.5071049Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5071136Z fn() 2022-11-23T03:11:33.5071525Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5071646Z return func(*args, **kwargs) 2022-11-23T03:11:33.5071900Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5072006Z self._dist_train() 2022-11-23T03:11:33.5072243Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5072554Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5072779Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5072913Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5073106Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5073245Z raise self.failureException(msg) 2022-11-23T03:11:33.5073376Z AssertionError: False is not true 2022-11-23T03:11:33.5073510Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5073616Z dist init r=1, world=2 2022-11-23T03:11:33.5074755Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5074865Z warnings.warn( 2022-11-23T03:11:33.5075069Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5075241Z Traceback (most recent call last): 2022-11-23T03:11:33.5075642Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5075762Z getattr(self, test_name)() 2022-11-23T03:11:33.5076159Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5076253Z fn() 2022-11-23T03:11:33.5076649Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5076773Z return func(*args, **kwargs) 2022-11-23T03:11:33.5077031Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5077138Z self._dist_train() 2022-11-23T03:11:33.5077379Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5077641Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5077863Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5078000Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5078190Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5078328Z raise self.failureException(msg) 2022-11-23T03:11:33.5078458Z AssertionError: False is not true 2022-11-23T03:11:33.5078587Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5078676Z dist init r=0, world=2 2022-11-23T03:11:33.5078881Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5078979Z ERROR (3.832s) 2022-11-23T03:11:33.5079413Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 36 2022-11-23T03:11:33.5079541Z Traceback (most recent call last): 2022-11-23T03:11:33.5079939Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5080058Z self._join_processes(fn) 2022-11-23T03:11:33.5080555Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5080703Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5081124Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5081249Z raise RuntimeError(error) 2022-11-23T03:11:33.5081438Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5081632Z Traceback (most recent call last): 2022-11-23T03:11:33.5082033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5082156Z getattr(self, test_name)() 2022-11-23T03:11:33.5082550Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5082641Z fn() 2022-11-23T03:11:33.5083013Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5083136Z return func(*args, **kwargs) 2022-11-23T03:11:33.5083387Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5083493Z self._dist_train() 2022-11-23T03:11:33.5083737Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5083996Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5084228Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5084369Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5084561Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5084781Z raise self.failureException(msg) 2022-11-23T03:11:33.5084915Z AssertionError: False is not true 2022-11-23T03:11:33.5084923Z 2022-11-23T03:11:33.5085084Z Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5085214Z Traceback (most recent call last): 2022-11-23T03:11:33.5085617Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5085735Z getattr(self, test_name)() 2022-11-23T03:11:33.5086130Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5086223Z fn() 2022-11-23T03:11:33.5086598Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5086721Z return func(*args, **kwargs) 2022-11-23T03:11:33.5086981Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5087090Z self._dist_train() 2022-11-23T03:11:33.5087331Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5087586Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5087815Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5087952Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5088148Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5088288Z raise self.failureException(msg) 2022-11-23T03:11:33.5088422Z AssertionError: False is not true 2022-11-23T03:11:33.5088432Z 2022-11-23T03:11:33.5088438Z 2022-11-23T03:11:33.5088444Z 2022-11-23T03:11:33.5088687Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47718 2022-11-23T03:11:33.5088912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47719 2022-11-23T03:11:33.5089322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5089499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5089913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5090110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5090363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5090765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5090987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5091403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5091584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5091841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5092278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5092707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5092942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5093175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5094377Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5094487Z warnings.warn( 2022-11-23T03:11:33.5094695Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5094827Z Traceback (most recent call last): 2022-11-23T03:11:33.5095227Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5095349Z getattr(self, test_name)() 2022-11-23T03:11:33.5095744Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5095839Z fn() 2022-11-23T03:11:33.5096228Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5096348Z return func(*args, **kwargs) 2022-11-23T03:11:33.5096606Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5096714Z self._dist_train() 2022-11-23T03:11:33.5096955Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5097209Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5097438Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5097574Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5097753Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5097895Z raise self.failureException(msg) 2022-11-23T03:11:33.5098025Z AssertionError: False is not true 2022-11-23T03:11:33.5098160Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5099305Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5099417Z warnings.warn( 2022-11-23T03:11:33.5099622Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5099748Z Traceback (most recent call last): 2022-11-23T03:11:33.5100149Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5100315Z getattr(self, test_name)() 2022-11-23T03:11:33.5100712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5100804Z fn() 2022-11-23T03:11:33.5101196Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5101309Z return func(*args, **kwargs) 2022-11-23T03:11:33.5101561Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5101668Z self._dist_train() 2022-11-23T03:11:33.5101906Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5102160Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5102388Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5102512Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5102710Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5102854Z raise self.failureException(msg) 2022-11-23T03:11:33.5102989Z AssertionError: False is not true 2022-11-23T03:11:33.5103128Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5103284Z dist init r=0, world=2 2022-11-23T03:11:33.5103392Z dist init r=1, world=2 2022-11-23T03:11:33.5103599Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5103701Z ERROR (3.731s) 2022-11-23T03:11:33.5104142Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 35 2022-11-23T03:11:33.5104276Z Traceback (most recent call last): 2022-11-23T03:11:33.5104673Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5104795Z self._join_processes(fn) 2022-11-23T03:11:33.5105210Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5105356Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5105783Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5105891Z raise RuntimeError(error) 2022-11-23T03:11:33.5106084Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5106219Z Traceback (most recent call last): 2022-11-23T03:11:33.5106616Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5106738Z getattr(self, test_name)() 2022-11-23T03:11:33.5107129Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5107223Z fn() 2022-11-23T03:11:33.5107618Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5107735Z return func(*args, **kwargs) 2022-11-23T03:11:33.5107989Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5108092Z self._dist_train() 2022-11-23T03:11:33.5108336Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5108595Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5108826Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5108966Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5109162Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5109305Z raise self.failureException(msg) 2022-11-23T03:11:33.5109422Z AssertionError: False is not true 2022-11-23T03:11:33.5109484Z 2022-11-23T03:11:33.5109509Z 2022-11-23T03:11:33.5109515Z 2022-11-23T03:11:33.5109750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47849 2022-11-23T03:11:33.5109958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47850 2022-11-23T03:11:33.5110377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5110558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5110976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5111171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5111429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5111835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5112020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5112432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5112673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5112927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5113369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5113804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5114042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5114280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5115430Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5115543Z warnings.warn( 2022-11-23T03:11:33.5115753Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5115889Z Traceback (most recent call last): 2022-11-23T03:11:33.5116290Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5116413Z getattr(self, test_name)() 2022-11-23T03:11:33.5116811Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5116908Z fn() 2022-11-23T03:11:33.5117282Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5117399Z return func(*args, **kwargs) 2022-11-23T03:11:33.5117665Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5117774Z self._dist_train() 2022-11-23T03:11:33.5118012Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5118271Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5118503Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5118640Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5118834Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5119024Z raise self.failureException(msg) 2022-11-23T03:11:33.5119155Z AssertionError: False is not true 2022-11-23T03:11:33.5119289Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5120528Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5120639Z warnings.warn( 2022-11-23T03:11:33.5120846Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5120977Z Traceback (most recent call last): 2022-11-23T03:11:33.5121380Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5121502Z getattr(self, test_name)() 2022-11-23T03:11:33.5121900Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5121973Z fn() 2022-11-23T03:11:33.5122425Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5122544Z return func(*args, **kwargs) 2022-11-23T03:11:33.5122796Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5122905Z self._dist_train() 2022-11-23T03:11:33.5123146Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5123405Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5123634Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5123778Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5123969Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5124112Z raise self.failureException(msg) 2022-11-23T03:11:33.5124246Z AssertionError: False is not true 2022-11-23T03:11:33.5124377Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5124488Z dist init r=0, world=2 2022-11-23T03:11:33.5124589Z dist init r=1, world=2 2022-11-23T03:11:33.5124795Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5124878Z ERROR (3.631s) 2022-11-23T03:11:33.5125322Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 34 2022-11-23T03:11:33.5125456Z Traceback (most recent call last): 2022-11-23T03:11:33.5125851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5125974Z self._join_processes(fn) 2022-11-23T03:11:33.5126386Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5126533Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5126948Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5127075Z raise RuntimeError(error) 2022-11-23T03:11:33.5127264Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5127397Z Traceback (most recent call last): 2022-11-23T03:11:33.5127792Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5127909Z getattr(self, test_name)() 2022-11-23T03:11:33.5128303Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5128394Z fn() 2022-11-23T03:11:33.5128781Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5128975Z return func(*args, **kwargs) 2022-11-23T03:11:33.5129216Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5129320Z self._dist_train() 2022-11-23T03:11:33.5129564Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5129824Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5130049Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5130185Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5130380Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5130518Z raise self.failureException(msg) 2022-11-23T03:11:33.5130650Z AssertionError: False is not true 2022-11-23T03:11:33.5130661Z 2022-11-23T03:11:33.5130666Z 2022-11-23T03:11:33.5130672Z 2022-11-23T03:11:33.5130898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47980 2022-11-23T03:11:33.5131117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47981 2022-11-23T03:11:33.5131568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5131748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5132166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5132359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5132612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5133017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5133198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5133605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5133799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5134051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5134494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5134913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5135146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5135378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5136528Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5136638Z warnings.warn( 2022-11-23T03:11:33.5136843Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5136975Z Traceback (most recent call last): 2022-11-23T03:11:33.5137376Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5137495Z getattr(self, test_name)() 2022-11-23T03:11:33.5137889Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5138029Z fn() 2022-11-23T03:11:33.5138424Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5138544Z return func(*args, **kwargs) 2022-11-23T03:11:33.5138802Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5138910Z self._dist_train() 2022-11-23T03:11:33.5139154Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5139412Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5139642Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5139778Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5139974Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5140116Z raise self.failureException(msg) 2022-11-23T03:11:33.5140231Z AssertionError: False is not true 2022-11-23T03:11:33.5140364Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5140469Z dist init r=1, world=2 2022-11-23T03:11:33.5141665Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5141781Z warnings.warn( 2022-11-23T03:11:33.5141987Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5142115Z Traceback (most recent call last): 2022-11-23T03:11:33.5142517Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5142635Z getattr(self, test_name)() 2022-11-23T03:11:33.5143028Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5143120Z fn() 2022-11-23T03:11:33.5143509Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5143629Z return func(*args, **kwargs) 2022-11-23T03:11:33.5143886Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5143988Z self._dist_train() 2022-11-23T03:11:33.5144231Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5144484Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5144714Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5144854Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5145032Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5145176Z raise self.failureException(msg) 2022-11-23T03:11:33.5145309Z AssertionError: False is not true 2022-11-23T03:11:33.5145443Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5145551Z dist init r=0, world=2 2022-11-23T03:11:33.5145755Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5145852Z ERROR (3.730s) 2022-11-23T03:11:33.5146287Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 33 2022-11-23T03:11:33.5146417Z Traceback (most recent call last): 2022-11-23T03:11:33.5146809Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5146928Z self._join_processes(fn) 2022-11-23T03:11:33.5147390Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5147534Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5147952Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5148079Z raise RuntimeError(error) 2022-11-23T03:11:33.5148270Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5148384Z Traceback (most recent call last): 2022-11-23T03:11:33.5148774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5148892Z getattr(self, test_name)() 2022-11-23T03:11:33.5149282Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5149374Z fn() 2022-11-23T03:11:33.5149762Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5149886Z return func(*args, **kwargs) 2022-11-23T03:11:33.5150138Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5150241Z self._dist_train() 2022-11-23T03:11:33.5150522Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5150781Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5151011Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5151147Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5151342Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5151483Z raise self.failureException(msg) 2022-11-23T03:11:33.5151610Z AssertionError: False is not true 2022-11-23T03:11:33.5151618Z 2022-11-23T03:11:33.5151628Z 2022-11-23T03:11:33.5151633Z 2022-11-23T03:11:33.5151853Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48111 2022-11-23T03:11:33.5152073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48112 2022-11-23T03:11:33.5152480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5152641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5153051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5153244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5153495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5153897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5154073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5154486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5154682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5154936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5155370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5155799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5156031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5156266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5157487Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5157594Z warnings.warn( 2022-11-23T03:11:33.5157800Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5157937Z Traceback (most recent call last): 2022-11-23T03:11:33.5158339Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5158461Z getattr(self, test_name)() 2022-11-23T03:11:33.5158864Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5158963Z fn() 2022-11-23T03:11:33.5159355Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5159479Z return func(*args, **kwargs) 2022-11-23T03:11:33.5159768Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5159877Z self._dist_train() 2022-11-23T03:11:33.5160119Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5160453Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5160681Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5160819Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5161015Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5161159Z raise self.failureException(msg) 2022-11-23T03:11:33.5172584Z AssertionError: False is not true 2022-11-23T03:11:33.5172784Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5172897Z dist init r=0, world=2 2022-11-23T03:11:33.5174137Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5174246Z warnings.warn( 2022-11-23T03:11:33.5174458Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5174588Z Traceback (most recent call last): 2022-11-23T03:11:33.5174997Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5175106Z getattr(self, test_name)() 2022-11-23T03:11:33.5175507Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5175598Z fn() 2022-11-23T03:11:33.5175995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5176117Z return func(*args, **kwargs) 2022-11-23T03:11:33.5176372Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5176475Z self._dist_train() 2022-11-23T03:11:33.5176715Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5176973Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5177199Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5177516Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5177713Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5177849Z raise self.failureException(msg) 2022-11-23T03:11:33.5177979Z AssertionError: False is not true 2022-11-23T03:11:33.5178105Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5178211Z dist init r=1, world=2 2022-11-23T03:11:33.5178400Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5178495Z ERROR (3.831s) 2022-11-23T03:11:33.5178935Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 32 2022-11-23T03:11:33.5179066Z Traceback (most recent call last): 2022-11-23T03:11:33.5179461Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5179579Z self._join_processes(fn) 2022-11-23T03:11:33.5179994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5180139Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5180560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5180745Z raise RuntimeError(error) 2022-11-23T03:11:33.5180937Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5181065Z Traceback (most recent call last): 2022-11-23T03:11:33.5181460Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5181578Z getattr(self, test_name)() 2022-11-23T03:11:33.5181975Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5182059Z fn() 2022-11-23T03:11:33.5182445Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5182551Z return func(*args, **kwargs) 2022-11-23T03:11:33.5182802Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5182906Z self._dist_train() 2022-11-23T03:11:33.5183147Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5183407Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5183637Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5183774Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5183963Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5184103Z raise self.failureException(msg) 2022-11-23T03:11:33.5184235Z AssertionError: False is not true 2022-11-23T03:11:33.5184244Z 2022-11-23T03:11:33.5184253Z 2022-11-23T03:11:33.5184259Z 2022-11-23T03:11:33.5184485Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48242 2022-11-23T03:11:33.5184702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48243 2022-11-23T03:11:33.5185114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5185289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5185700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5185894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5186146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5186547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5186768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5187178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5187368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5187623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5188045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5188486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5188721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5188953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5190142Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5190249Z warnings.warn( 2022-11-23T03:11:33.5190454Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5190580Z Traceback (most recent call last): 2022-11-23T03:11:33.5190982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5191101Z getattr(self, test_name)() 2022-11-23T03:11:33.5191494Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5191585Z fn() 2022-11-23T03:11:33.5191973Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5192092Z return func(*args, **kwargs) 2022-11-23T03:11:33.5192345Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5192442Z self._dist_train() 2022-11-23T03:11:33.5192681Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5192937Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5193161Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5193297Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5193486Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5193613Z raise self.failureException(msg) 2022-11-23T03:11:33.5193745Z AssertionError: False is not true 2022-11-23T03:11:33.5193878Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5193982Z dist init r=1, world=2 2022-11-23T03:11:33.5195131Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5195240Z warnings.warn( 2022-11-23T03:11:33.5195447Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5195578Z Traceback (most recent call last): 2022-11-23T03:11:33.5195976Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5196285Z getattr(self, test_name)() 2022-11-23T03:11:33.5196676Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5196761Z fn() 2022-11-23T03:11:33.5197151Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5197268Z return func(*args, **kwargs) 2022-11-23T03:11:33.5197517Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5197619Z self._dist_train() 2022-11-23T03:11:33.5197856Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5198104Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5198326Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5198446Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5198642Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5198780Z raise self.failureException(msg) 2022-11-23T03:11:33.5198910Z AssertionError: False is not true 2022-11-23T03:11:33.5199042Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5199193Z dist init r=0, world=2 2022-11-23T03:11:33.5199397Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5199496Z ERROR (3.636s) 2022-11-23T03:11:33.5199930Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 31 2022-11-23T03:11:33.5200057Z Traceback (most recent call last): 2022-11-23T03:11:33.5200529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5200643Z self._join_processes(fn) 2022-11-23T03:11:33.5201052Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5201196Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5201617Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5201732Z raise RuntimeError(error) 2022-11-23T03:11:33.5201908Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5202034Z Traceback (most recent call last): 2022-11-23T03:11:33.5202419Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5202537Z getattr(self, test_name)() 2022-11-23T03:11:33.5202925Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5203013Z fn() 2022-11-23T03:11:33.5203400Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5203520Z return func(*args, **kwargs) 2022-11-23T03:11:33.5203774Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5203875Z self._dist_train() 2022-11-23T03:11:33.5204113Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5204362Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5204591Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5204722Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5204911Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5205043Z raise self.failureException(msg) 2022-11-23T03:11:33.5205169Z AssertionError: False is not true 2022-11-23T03:11:33.5205177Z 2022-11-23T03:11:33.5205183Z 2022-11-23T03:11:33.5205245Z 2022-11-23T03:11:33.5205466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48373 2022-11-23T03:11:33.5205682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48374 2022-11-23T03:11:33.5206075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5206255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5206667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5206859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5207113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5207514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5207687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5208097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5208289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5208588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5209025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5209459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5209694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5209923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5211075Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5211181Z warnings.warn( 2022-11-23T03:11:33.5211384Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5211508Z Traceback (most recent call last): 2022-11-23T03:11:33.5211903Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5212017Z getattr(self, test_name)() 2022-11-23T03:11:33.5212409Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5212500Z fn() 2022-11-23T03:11:33.5212888Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5212993Z return func(*args, **kwargs) 2022-11-23T03:11:33.5213241Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5213346Z self._dist_train() 2022-11-23T03:11:33.5213584Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5213839Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5214067Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5214203Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5214395Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5214534Z raise self.failureException(msg) 2022-11-23T03:11:33.5214717Z AssertionError: False is not true 2022-11-23T03:11:33.5214846Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5214944Z dist init r=0, world=2 2022-11-23T03:11:33.5216093Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5216194Z warnings.warn( 2022-11-23T03:11:33.5216393Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5216520Z Traceback (most recent call last): 2022-11-23T03:11:33.5216912Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5217029Z getattr(self, test_name)() 2022-11-23T03:11:33.5217421Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5217495Z fn() 2022-11-23T03:11:33.5217884Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5218044Z return func(*args, **kwargs) 2022-11-23T03:11:33.5218298Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5218401Z self._dist_train() 2022-11-23T03:11:33.5218649Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5218908Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5219136Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5219270Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5219465Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5219605Z raise self.failureException(msg) 2022-11-23T03:11:33.5219735Z AssertionError: False is not true 2022-11-23T03:11:33.5219867Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5219967Z dist init r=1, world=2 2022-11-23T03:11:33.5220171Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5220269Z ERROR (3.932s) 2022-11-23T03:11:33.5220689Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 30 2022-11-23T03:11:33.5220817Z Traceback (most recent call last): 2022-11-23T03:11:33.5221212Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5221333Z self._join_processes(fn) 2022-11-23T03:11:33.5221745Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5221895Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5222315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5222435Z raise RuntimeError(error) 2022-11-23T03:11:33.5222627Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5222755Z Traceback (most recent call last): 2022-11-23T03:11:33.5223151Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5223266Z getattr(self, test_name)() 2022-11-23T03:11:33.5223655Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5223740Z fn() 2022-11-23T03:11:33.5224129Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5224300Z return func(*args, **kwargs) 2022-11-23T03:11:33.5224550Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5224653Z self._dist_train() 2022-11-23T03:11:33.5224879Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5225137Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5225365Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5225503Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5225697Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5225836Z raise self.failureException(msg) 2022-11-23T03:11:33.5225970Z AssertionError: False is not true 2022-11-23T03:11:33.5225977Z 2022-11-23T03:11:33.5226137Z Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5226269Z Traceback (most recent call last): 2022-11-23T03:11:33.5226662Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5226776Z getattr(self, test_name)() 2022-11-23T03:11:33.5227210Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5227303Z fn() 2022-11-23T03:11:33.5227690Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5227809Z return func(*args, **kwargs) 2022-11-23T03:11:33.5228061Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5228162Z self._dist_train() 2022-11-23T03:11:33.5228385Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5228633Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5228866Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5229000Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5229191Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5229335Z raise self.failureException(msg) 2022-11-23T03:11:33.5229466Z AssertionError: False is not true 2022-11-23T03:11:33.5229473Z 2022-11-23T03:11:33.5229479Z 2022-11-23T03:11:33.5229485Z 2022-11-23T03:11:33.5229711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48504 2022-11-23T03:11:33.5229936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48505 2022-11-23T03:11:33.5230341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5230517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5230933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5231127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5231379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5231779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5231952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5232359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5232551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5232803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5233286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5233715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5233946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5234164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5235319Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5235422Z warnings.warn( 2022-11-23T03:11:33.5235613Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5235743Z Traceback (most recent call last): 2022-11-23T03:11:33.5236142Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5236337Z getattr(self, test_name)() 2022-11-23T03:11:33.5236741Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5236836Z fn() 2022-11-23T03:11:33.5237230Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5237352Z return func(*args, **kwargs) 2022-11-23T03:11:33.5237603Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5237705Z self._dist_train() 2022-11-23T03:11:33.5237944Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5238202Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5238427Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5238561Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5238758Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5238891Z raise self.failureException(msg) 2022-11-23T03:11:33.5239023Z AssertionError: False is not true 2022-11-23T03:11:33.5239141Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5239241Z dist init r=1, world=2 2022-11-23T03:11:33.5240467Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5240578Z warnings.warn( 2022-11-23T03:11:33.5240783Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5240915Z Traceback (most recent call last): 2022-11-23T03:11:33.5241316Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5241438Z getattr(self, test_name)() 2022-11-23T03:11:33.5241828Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5241918Z fn() 2022-11-23T03:11:33.5242305Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5242421Z return func(*args, **kwargs) 2022-11-23T03:11:33.5242737Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5242838Z self._dist_train() 2022-11-23T03:11:33.5243073Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5243329Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5243550Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5243681Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5243872Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5243998Z raise self.failureException(msg) 2022-11-23T03:11:33.5244131Z AssertionError: False is not true 2022-11-23T03:11:33.5244262Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5244367Z dist init r=0, world=2 2022-11-23T03:11:33.5244573Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5244675Z ERROR (3.631s) 2022-11-23T03:11:33.5245108Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 29 2022-11-23T03:11:33.5245237Z Traceback (most recent call last): 2022-11-23T03:11:33.5245681Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5245802Z self._join_processes(fn) 2022-11-23T03:11:33.5246213Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5246359Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5246772Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5246893Z raise RuntimeError(error) 2022-11-23T03:11:33.5247081Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5247213Z Traceback (most recent call last): 2022-11-23T03:11:33.5247594Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5247706Z getattr(self, test_name)() 2022-11-23T03:11:33.5248097Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5248189Z fn() 2022-11-23T03:11:33.5248578Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5248699Z return func(*args, **kwargs) 2022-11-23T03:11:33.5248949Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5249055Z self._dist_train() 2022-11-23T03:11:33.5249292Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5249547Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5249777Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5249911Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5250104Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5250247Z raise self.failureException(msg) 2022-11-23T03:11:33.5250379Z AssertionError: False is not true 2022-11-23T03:11:33.5250388Z 2022-11-23T03:11:33.5250549Z Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5250671Z Traceback (most recent call last): 2022-11-23T03:11:33.5251057Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5251159Z getattr(self, test_name)() 2022-11-23T03:11:33.5251543Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5251630Z fn() 2022-11-23T03:11:33.5252067Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5252185Z return func(*args, **kwargs) 2022-11-23T03:11:33.5252434Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5252539Z self._dist_train() 2022-11-23T03:11:33.5252772Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5253020Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5253241Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5253376Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5253569Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5253709Z raise self.failureException(msg) 2022-11-23T03:11:33.5253840Z AssertionError: False is not true 2022-11-23T03:11:33.5253850Z 2022-11-23T03:11:33.5253856Z 2022-11-23T03:11:33.5253862Z 2022-11-23T03:11:33.5254087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48635 2022-11-23T03:11:33.5254308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48636 2022-11-23T03:11:33.5254756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5254934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5255345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5255522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5255771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5256170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5256349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5256759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5256949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5257200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5257636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5258067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5258301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5258530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5259686Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5259793Z warnings.warn( 2022-11-23T03:11:33.5259999Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5260130Z Traceback (most recent call last): 2022-11-23T03:11:33.5260527Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5260645Z getattr(self, test_name)() 2022-11-23T03:11:33.5261036Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5261172Z fn() 2022-11-23T03:11:33.5261561Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5261681Z return func(*args, **kwargs) 2022-11-23T03:11:33.5261934Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5262040Z self._dist_train() 2022-11-23T03:11:33.5262265Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5262515Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5262737Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5262873Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5263066Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5263209Z raise self.failureException(msg) 2022-11-23T03:11:33.5263341Z AssertionError: False is not true 2022-11-23T03:11:33.5263470Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5264653Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5264760Z warnings.warn( 2022-11-23T03:11:33.5264962Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5265089Z Traceback (most recent call last): 2022-11-23T03:11:33.5265482Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5265602Z getattr(self, test_name)() 2022-11-23T03:11:33.5265995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5266081Z fn() 2022-11-23T03:11:33.5266469Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5266585Z return func(*args, **kwargs) 2022-11-23T03:11:33.5266833Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5266921Z self._dist_train() 2022-11-23T03:11:33.5267159Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5267413Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5267639Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5267777Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5267970Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5268114Z raise self.failureException(msg) 2022-11-23T03:11:33.5268246Z AssertionError: False is not true 2022-11-23T03:11:33.5268378Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5268483Z dist init r=1, world=2 2022-11-23T03:11:33.5268580Z dist init r=0, world=2 2022-11-23T03:11:33.5268784Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5268883Z ERROR (3.633s) 2022-11-23T03:11:33.5269318Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 28 2022-11-23T03:11:33.5269444Z Traceback (most recent call last): 2022-11-23T03:11:33.5269832Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5269936Z self._join_processes(fn) 2022-11-23T03:11:33.5270400Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5270538Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5270953Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5271078Z raise RuntimeError(error) 2022-11-23T03:11:33.5271266Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5271396Z Traceback (most recent call last): 2022-11-23T03:11:33.5271784Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5271901Z getattr(self, test_name)() 2022-11-23T03:11:33.5272292Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5272385Z fn() 2022-11-23T03:11:33.5272775Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5272895Z return func(*args, **kwargs) 2022-11-23T03:11:33.5273146Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5273295Z self._dist_train() 2022-11-23T03:11:33.5273537Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5273793Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5274004Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5274137Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5274330Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5274468Z raise self.failureException(msg) 2022-11-23T03:11:33.5274595Z AssertionError: False is not true 2022-11-23T03:11:33.5274606Z 2022-11-23T03:11:33.5274613Z 2022-11-23T03:11:33.5274619Z 2022-11-23T03:11:33.5274849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48766 2022-11-23T03:11:33.5275065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48767 2022-11-23T03:11:33.5275470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5275642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5276049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5276243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5276498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5276899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5277076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5277485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5277679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5277933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5278377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5278807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5279039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5279270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5280556Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5280663Z warnings.warn( 2022-11-23T03:11:33.5280864Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5280991Z Traceback (most recent call last): 2022-11-23T03:11:33.5281392Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5281495Z getattr(self, test_name)() 2022-11-23T03:11:33.5281890Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5281981Z fn() 2022-11-23T03:11:33.5282372Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5282492Z return func(*args, **kwargs) 2022-11-23T03:11:33.5282801Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5282909Z self._dist_train() 2022-11-23T03:11:33.5283148Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5283405Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5283630Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5283764Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5283957Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5284099Z raise self.failureException(msg) 2022-11-23T03:11:33.5284226Z AssertionError: False is not true 2022-11-23T03:11:33.5284354Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5285500Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5285601Z warnings.warn( 2022-11-23T03:11:33.5285804Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5285933Z Traceback (most recent call last): 2022-11-23T03:11:33.5286315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5286434Z getattr(self, test_name)() 2022-11-23T03:11:33.5286824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5286915Z fn() 2022-11-23T03:11:33.5287307Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5287428Z return func(*args, **kwargs) 2022-11-23T03:11:33.5287683Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5287784Z self._dist_train() 2022-11-23T03:11:33.5288021Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5288275Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5288506Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5288695Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5288887Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5289024Z raise self.failureException(msg) 2022-11-23T03:11:33.5289153Z AssertionError: False is not true 2022-11-23T03:11:33.5289281Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5289374Z dist init r=1, world=2 2022-11-23T03:11:33.5289478Z dist init r=0, world=2 2022-11-23T03:11:33.5289683Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5289779Z ERROR (3.832s) 2022-11-23T03:11:33.5290217Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 27 2022-11-23T03:11:33.5290344Z Traceback (most recent call last): 2022-11-23T03:11:33.5290732Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5290853Z self._join_processes(fn) 2022-11-23T03:11:33.5291265Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5291408Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5291825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5291991Z raise RuntimeError(error) 2022-11-23T03:11:33.5292181Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5292309Z Traceback (most recent call last): 2022-11-23T03:11:33.5292699Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5292816Z getattr(self, test_name)() 2022-11-23T03:11:33.5293197Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5293272Z fn() 2022-11-23T03:11:33.5293658Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5293776Z return func(*args, **kwargs) 2022-11-23T03:11:33.5294033Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5294139Z self._dist_train() 2022-11-23T03:11:33.5294380Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5294631Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5294854Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5294986Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5295176Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5295316Z raise self.failureException(msg) 2022-11-23T03:11:33.5295445Z AssertionError: False is not true 2022-11-23T03:11:33.5295456Z 2022-11-23T03:11:33.5295462Z 2022-11-23T03:11:33.5295467Z 2022-11-23T03:11:33.5295693Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48897 2022-11-23T03:11:33.5295912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48898 2022-11-23T03:11:33.5296318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5296498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5296908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5297102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5297350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5297752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5297960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5298371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5298556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5298809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5299240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5299674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5299907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5300137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5301330Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5301440Z warnings.warn( 2022-11-23T03:11:33.5301648Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5301777Z Traceback (most recent call last): 2022-11-23T03:11:33.5302175Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5302291Z getattr(self, test_name)() 2022-11-23T03:11:33.5302687Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5302780Z fn() 2022-11-23T03:11:33.5303165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5303280Z return func(*args, **kwargs) 2022-11-23T03:11:33.5303533Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5303635Z self._dist_train() 2022-11-23T03:11:33.5303873Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5304125Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5304339Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5304470Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5304663Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5304807Z raise self.failureException(msg) 2022-11-23T03:11:33.5304936Z AssertionError: False is not true 2022-11-23T03:11:33.5305070Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5306206Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5306316Z warnings.warn( 2022-11-23T03:11:33.5306520Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5306646Z Traceback (most recent call last): 2022-11-23T03:11:33.5307042Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5307229Z getattr(self, test_name)() 2022-11-23T03:11:33.5307625Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5307711Z fn() 2022-11-23T03:11:33.5308102Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5308221Z return func(*args, **kwargs) 2022-11-23T03:11:33.5308471Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5308570Z self._dist_train() 2022-11-23T03:11:33.5308809Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5309061Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5309271Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5309406Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5309601Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5309742Z raise self.failureException(msg) 2022-11-23T03:11:33.5309874Z AssertionError: False is not true 2022-11-23T03:11:33.5310006Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5310153Z dist init r=0, world=2 2022-11-23T03:11:33.5310255Z dist init r=1, world=2 2022-11-23T03:11:33.5310456Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5310553Z ERROR (3.733s) 2022-11-23T03:11:33.5310987Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 26 2022-11-23T03:11:33.5311114Z Traceback (most recent call last): 2022-11-23T03:11:33.5311504Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5311620Z self._join_processes(fn) 2022-11-23T03:11:33.5312030Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5312158Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5312568Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5312689Z raise RuntimeError(error) 2022-11-23T03:11:33.5312876Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5313003Z Traceback (most recent call last): 2022-11-23T03:11:33.5313396Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5313511Z getattr(self, test_name)() 2022-11-23T03:11:33.5313903Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5313992Z fn() 2022-11-23T03:11:33.5314381Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5314506Z return func(*args, **kwargs) 2022-11-23T03:11:33.5314758Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5314862Z self._dist_train() 2022-11-23T03:11:33.5315103Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5315359Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5315582Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5315715Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5315903Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5316028Z raise self.failureException(msg) 2022-11-23T03:11:33.5316158Z AssertionError: False is not true 2022-11-23T03:11:33.5316165Z 2022-11-23T03:11:33.5316220Z 2022-11-23T03:11:33.5316225Z 2022-11-23T03:11:33.5316448Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49028 2022-11-23T03:11:33.5316662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49029 2022-11-23T03:11:33.5317070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5317243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5317656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5317845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5318093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5318492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5318673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5319084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5319272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5319572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5320015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5320550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5320787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5321018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5322171Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5322276Z warnings.warn( 2022-11-23T03:11:33.5322477Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5322605Z Traceback (most recent call last): 2022-11-23T03:11:33.5322999Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5323112Z getattr(self, test_name)() 2022-11-23T03:11:33.5323504Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5323582Z fn() 2022-11-23T03:11:33.5323968Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5324086Z return func(*args, **kwargs) 2022-11-23T03:11:33.5324343Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5324450Z self._dist_train() 2022-11-23T03:11:33.5324687Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5324941Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5325166Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5325301Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5325494Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5325631Z raise self.failureException(msg) 2022-11-23T03:11:33.5325817Z AssertionError: False is not true 2022-11-23T03:11:33.5325946Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5326046Z dist init r=0, world=2 2022-11-23T03:11:33.5327185Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5327289Z warnings.warn( 2022-11-23T03:11:33.5327488Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5327614Z Traceback (most recent call last): 2022-11-23T03:11:33.5328008Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5328114Z getattr(self, test_name)() 2022-11-23T03:11:33.5328507Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5328599Z fn() 2022-11-23T03:11:33.5329044Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5329166Z return func(*args, **kwargs) 2022-11-23T03:11:33.5329417Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5329521Z self._dist_train() 2022-11-23T03:11:33.5329757Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5330009Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5330232Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5330369Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5330557Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5330690Z raise self.failureException(msg) 2022-11-23T03:11:33.5330818Z AssertionError: False is not true 2022-11-23T03:11:33.5330944Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5331049Z dist init r=1, world=2 2022-11-23T03:11:33.5331237Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5331333Z ERROR (3.931s) 2022-11-23T03:11:33.5331763Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 25 2022-11-23T03:11:33.5331890Z Traceback (most recent call last): 2022-11-23T03:11:33.5332283Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5332404Z self._join_processes(fn) 2022-11-23T03:11:33.5332813Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5332961Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5333379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5333503Z raise RuntimeError(error) 2022-11-23T03:11:33.5333691Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5333820Z Traceback (most recent call last): 2022-11-23T03:11:33.5334207Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5334320Z getattr(self, test_name)() 2022-11-23T03:11:33.5334709Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5334796Z fn() 2022-11-23T03:11:33.5335174Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5335329Z return func(*args, **kwargs) 2022-11-23T03:11:33.5335579Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5335682Z self._dist_train() 2022-11-23T03:11:33.5335921Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5336171Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5336394Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5336526Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5336718Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5336855Z raise self.failureException(msg) 2022-11-23T03:11:33.5336981Z AssertionError: False is not true 2022-11-23T03:11:33.5336989Z 2022-11-23T03:11:33.5336998Z 2022-11-23T03:11:33.5337004Z 2022-11-23T03:11:33.5337228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49159 2022-11-23T03:11:33.5337445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49160 2022-11-23T03:11:33.5337892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5338072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5338485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5338675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5338929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5339333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5339509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5339916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5340107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5340347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5340777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5341203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5341432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5341664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5342813Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5342919Z warnings.warn( 2022-11-23T03:11:33.5343121Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5343254Z Traceback (most recent call last): 2022-11-23T03:11:33.5343654Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5343775Z getattr(self, test_name)() 2022-11-23T03:11:33.5344167Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5344306Z fn() 2022-11-23T03:11:33.5344696Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5344815Z return func(*args, **kwargs) 2022-11-23T03:11:33.5345069Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5345172Z self._dist_train() 2022-11-23T03:11:33.5345409Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5345662Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5345888Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5346018Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5346206Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5346331Z raise self.failureException(msg) 2022-11-23T03:11:33.5346464Z AssertionError: False is not true 2022-11-23T03:11:33.5346593Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5347786Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5347895Z warnings.warn( 2022-11-23T03:11:33.5348100Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5348230Z Traceback (most recent call last): 2022-11-23T03:11:33.5348631Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5348747Z getattr(self, test_name)() 2022-11-23T03:11:33.5349142Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5349228Z fn() 2022-11-23T03:11:33.5349618Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5349736Z return func(*args, **kwargs) 2022-11-23T03:11:33.5349984Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5350083Z self._dist_train() 2022-11-23T03:11:33.5350318Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5350572Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5350798Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5350930Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5351110Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5351250Z raise self.failureException(msg) 2022-11-23T03:11:33.5351383Z AssertionError: False is not true 2022-11-23T03:11:33.5351516Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5351619Z dist init r=0, world=2 2022-11-23T03:11:33.5351724Z dist init r=1, world=2 2022-11-23T03:11:33.5351927Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5352022Z ERROR (3.728s) 2022-11-23T03:11:33.5352455Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 24 2022-11-23T03:11:33.5352583Z Traceback (most recent call last): 2022-11-23T03:11:33.5352965Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5353079Z self._join_processes(fn) 2022-11-23T03:11:33.5353486Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5353669Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5354090Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5354209Z raise RuntimeError(error) 2022-11-23T03:11:33.5354397Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5354527Z Traceback (most recent call last): 2022-11-23T03:11:33.5354913Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5355026Z getattr(self, test_name)() 2022-11-23T03:11:33.5355401Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5355487Z fn() 2022-11-23T03:11:33.5355870Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5355989Z return func(*args, **kwargs) 2022-11-23T03:11:33.5356240Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5356344Z self._dist_train() 2022-11-23T03:11:33.5356623Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5356878Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5357102Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5357236Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5357425Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5357567Z raise self.failureException(msg) 2022-11-23T03:11:33.5357696Z AssertionError: False is not true 2022-11-23T03:11:33.5357704Z 2022-11-23T03:11:33.5357711Z 2022-11-23T03:11:33.5357720Z 2022-11-23T03:11:33.5357946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49290 2022-11-23T03:11:33.5358163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49291 2022-11-23T03:11:33.5358567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5358742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5359163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5359353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5359592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5359989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5360226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5360636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5360828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5361081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5361522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5361956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5362192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5362429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5363580Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5363750Z warnings.warn( 2022-11-23T03:11:33.5363959Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5364092Z Traceback (most recent call last): 2022-11-23T03:11:33.5364492Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5364609Z getattr(self, test_name)() 2022-11-23T03:11:33.5365003Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5365095Z fn() 2022-11-23T03:11:33.5365481Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5365598Z return func(*args, **kwargs) 2022-11-23T03:11:33.5365850Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5366002Z self._dist_train() 2022-11-23T03:11:33.5366243Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5366485Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5366713Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5366852Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5367046Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5367190Z raise self.failureException(msg) 2022-11-23T03:11:33.5367325Z AssertionError: False is not true 2022-11-23T03:11:33.5367458Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5367563Z dist init r=0, world=2 2022-11-23T03:11:33.5368719Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5368827Z warnings.warn( 2022-11-23T03:11:33.5369036Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5369169Z Traceback (most recent call last): 2022-11-23T03:11:33.5369568Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5369689Z getattr(self, test_name)() 2022-11-23T03:11:33.5370073Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5370161Z fn() 2022-11-23T03:11:33.5370544Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5370668Z return func(*args, **kwargs) 2022-11-23T03:11:33.5370921Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5371011Z self._dist_train() 2022-11-23T03:11:33.5371250Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5371512Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5371740Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5371879Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5372121Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5372260Z raise self.failureException(msg) 2022-11-23T03:11:33.5372391Z AssertionError: False is not true 2022-11-23T03:11:33.5372520Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5372620Z dist init r=1, world=2 2022-11-23T03:11:33.5372826Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5372924Z ERROR (3.731s) 2022-11-23T03:11:33.5373358Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 23 2022-11-23T03:11:33.5373485Z Traceback (most recent call last): 2022-11-23T03:11:33.5373877Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5373987Z self._join_processes(fn) 2022-11-23T03:11:33.5374384Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5374525Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5374931Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5375051Z raise RuntimeError(error) 2022-11-23T03:11:33.5375329Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5375460Z Traceback (most recent call last): 2022-11-23T03:11:33.5375853Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5375971Z getattr(self, test_name)() 2022-11-23T03:11:33.5376362Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5376453Z fn() 2022-11-23T03:11:33.5376840Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5376966Z return func(*args, **kwargs) 2022-11-23T03:11:33.5377221Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5377328Z self._dist_train() 2022-11-23T03:11:33.5377572Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5377829Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5378053Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5378174Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5378364Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5378504Z raise self.failureException(msg) 2022-11-23T03:11:33.5378637Z AssertionError: False is not true 2022-11-23T03:11:33.5378645Z 2022-11-23T03:11:33.5378652Z 2022-11-23T03:11:33.5378658Z 2022-11-23T03:11:33.5378886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49421 2022-11-23T03:11:33.5379109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49422 2022-11-23T03:11:33.5379512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5379689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5380098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5380295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5380553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5380957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5381132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5381598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5381790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5382047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5382480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5382915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5383149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5383375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5384562Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5384674Z warnings.warn( 2022-11-23T03:11:33.5384877Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5385004Z Traceback (most recent call last): 2022-11-23T03:11:33.5385403Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5385520Z getattr(self, test_name)() 2022-11-23T03:11:33.5385898Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5385982Z fn() 2022-11-23T03:11:33.5386373Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5386494Z return func(*args, **kwargs) 2022-11-23T03:11:33.5386748Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5386855Z self._dist_train() 2022-11-23T03:11:33.5387093Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5387353Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5387579Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5387714Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5387907Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5388046Z raise self.failureException(msg) 2022-11-23T03:11:33.5388176Z AssertionError: False is not true 2022-11-23T03:11:33.5388308Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5388407Z dist init r=0, world=2 2022-11-23T03:11:33.5389546Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5389648Z warnings.warn( 2022-11-23T03:11:33.5389854Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5389969Z Traceback (most recent call last): 2022-11-23T03:11:33.5390370Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5390486Z getattr(self, test_name)() 2022-11-23T03:11:33.5390929Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5391017Z fn() 2022-11-23T03:11:33.5391412Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5391535Z return func(*args, **kwargs) 2022-11-23T03:11:33.5391784Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5391887Z self._dist_train() 2022-11-23T03:11:33.5392124Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5392380Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5392604Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5392735Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5392928Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5393067Z raise self.failureException(msg) 2022-11-23T03:11:33.5393200Z AssertionError: False is not true 2022-11-23T03:11:33.5393328Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5393418Z dist init r=1, world=2 2022-11-23T03:11:33.5393668Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5393766Z ERROR (3.731s) 2022-11-23T03:11:33.5394200Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 22 2022-11-23T03:11:33.5394331Z Traceback (most recent call last): 2022-11-23T03:11:33.5394720Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5394837Z self._join_processes(fn) 2022-11-23T03:11:33.5395249Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5395394Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5395811Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5395931Z raise RuntimeError(error) 2022-11-23T03:11:33.5396120Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5396247Z Traceback (most recent call last): 2022-11-23T03:11:33.5396636Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5396749Z getattr(self, test_name)() 2022-11-23T03:11:33.5397140Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5397225Z fn() 2022-11-23T03:11:33.5397599Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5397716Z return func(*args, **kwargs) 2022-11-23T03:11:33.5397968Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5398071Z self._dist_train() 2022-11-23T03:11:33.5398310Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5398561Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5398780Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5398908Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5399098Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5399236Z raise self.failureException(msg) 2022-11-23T03:11:33.5399367Z AssertionError: False is not true 2022-11-23T03:11:33.5399374Z 2022-11-23T03:11:33.5399380Z 2022-11-23T03:11:33.5399386Z 2022-11-23T03:11:33.5399653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49552 2022-11-23T03:11:33.5399873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49553 2022-11-23T03:11:33.5400362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5400545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5400959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5401153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5401405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5401809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5401983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5402380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5402569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5402877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5403320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5403751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5403986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5404215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5405361Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5405474Z warnings.warn( 2022-11-23T03:11:33.5405678Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5405808Z Traceback (most recent call last): 2022-11-23T03:11:33.5406205Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5406323Z getattr(self, test_name)() 2022-11-23T03:11:33.5406713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5406802Z fn() 2022-11-23T03:11:33.5407189Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5407305Z return func(*args, **kwargs) 2022-11-23T03:11:33.5407557Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5407661Z self._dist_train() 2022-11-23T03:11:33.5407896Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5408145Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5408367Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5408488Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5408676Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5408813Z raise self.failureException(msg) 2022-11-23T03:11:33.5408942Z AssertionError: False is not true 2022-11-23T03:11:33.5409125Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5410265Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5410373Z warnings.warn( 2022-11-23T03:11:33.5410574Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5410703Z Traceback (most recent call last): 2022-11-23T03:11:33.5411094Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5411214Z getattr(self, test_name)() 2022-11-23T03:11:33.5411603Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5411693Z fn() 2022-11-23T03:11:33.5412078Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5412195Z return func(*args, **kwargs) 2022-11-23T03:11:33.5412488Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5412590Z self._dist_train() 2022-11-23T03:11:33.5412826Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5413075Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5413303Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5413421Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5413619Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5413767Z raise self.failureException(msg) 2022-11-23T03:11:33.5413900Z AssertionError: False is not true 2022-11-23T03:11:33.5414034Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5414139Z dist init r=0, world=2 2022-11-23T03:11:33.5414238Z dist init r=1, world=2 2022-11-23T03:11:33.5414451Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5414545Z ERROR (3.633s) 2022-11-23T03:11:33.5414980Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 21 2022-11-23T03:11:33.5415107Z Traceback (most recent call last): 2022-11-23T03:11:33.5415497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5415614Z self._join_processes(fn) 2022-11-23T03:11:33.5416013Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5416158Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5416557Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5416678Z raise RuntimeError(error) 2022-11-23T03:11:33.5416864Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5416990Z Traceback (most recent call last): 2022-11-23T03:11:33.5417379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5417495Z getattr(self, test_name)() 2022-11-23T03:11:33.5417882Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5417970Z fn() 2022-11-23T03:11:33.5418354Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5418522Z return func(*args, **kwargs) 2022-11-23T03:11:33.5418773Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5418874Z self._dist_train() 2022-11-23T03:11:33.5419111Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5419367Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5419591Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5419725Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5419912Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5420038Z raise self.failureException(msg) 2022-11-23T03:11:33.5420160Z AssertionError: False is not true 2022-11-23T03:11:33.5420168Z 2022-11-23T03:11:33.5420183Z 2022-11-23T03:11:33.5420189Z 2022-11-23T03:11:33.5420399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49683 2022-11-23T03:11:33.5420620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49684 2022-11-23T03:11:33.5421020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5421242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5421661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5421857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5422106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5422509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5422681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5423092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5423282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5423533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5423973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5424404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5424636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5424862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5426026Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5426133Z warnings.warn( 2022-11-23T03:11:33.5426336Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5426465Z Traceback (most recent call last): 2022-11-23T03:11:33.5426864Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5426979Z getattr(self, test_name)() 2022-11-23T03:11:33.5427373Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5427468Z fn() 2022-11-23T03:11:33.5427840Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5428012Z return func(*args, **kwargs) 2022-11-23T03:11:33.5428266Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5428372Z self._dist_train() 2022-11-23T03:11:33.5428615Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5428868Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5429096Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5429231Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5429424Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5429564Z raise self.failureException(msg) 2022-11-23T03:11:33.5429691Z AssertionError: False is not true 2022-11-23T03:11:33.5429824Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5429923Z dist init r=0, world=2 2022-11-23T03:11:33.5431112Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5431218Z warnings.warn( 2022-11-23T03:11:33.5431420Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5431546Z Traceback (most recent call last): 2022-11-23T03:11:33.5431943Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5432063Z getattr(self, test_name)() 2022-11-23T03:11:33.5432443Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5432537Z fn() 2022-11-23T03:11:33.5432930Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5433056Z return func(*args, **kwargs) 2022-11-23T03:11:33.5433312Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5433419Z self._dist_train() 2022-11-23T03:11:33.5433663Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5433917Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5434142Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5434283Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5434475Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5434616Z raise self.failureException(msg) 2022-11-23T03:11:33.5434743Z AssertionError: False is not true 2022-11-23T03:11:33.5434871Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5434973Z dist init r=1, world=2 2022-11-23T03:11:33.5435179Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5435266Z ERROR (3.932s) 2022-11-23T03:11:33.5435707Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 20 2022-11-23T03:11:33.5435835Z Traceback (most recent call last): 2022-11-23T03:11:33.5436231Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5436351Z self._join_processes(fn) 2022-11-23T03:11:33.5436761Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5436957Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5437376Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5437497Z raise RuntimeError(error) 2022-11-23T03:11:33.5437690Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5437817Z Traceback (most recent call last): 2022-11-23T03:11:33.5438204Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5438320Z getattr(self, test_name)() 2022-11-23T03:11:33.5438706Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5438794Z fn() 2022-11-23T03:11:33.5439179Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5439298Z return func(*args, **kwargs) 2022-11-23T03:11:33.5439540Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5439644Z self._dist_train() 2022-11-23T03:11:33.5439884Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5440263Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5440496Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5440634Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5440827Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5440967Z raise self.failureException(msg) 2022-11-23T03:11:33.5441097Z AssertionError: False is not true 2022-11-23T03:11:33.5441105Z 2022-11-23T03:11:33.5441266Z Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5441395Z Traceback (most recent call last): 2022-11-23T03:11:33.5441798Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5441915Z getattr(self, test_name)() 2022-11-23T03:11:33.5442308Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5442400Z fn() 2022-11-23T03:11:33.5442788Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5442907Z return func(*args, **kwargs) 2022-11-23T03:11:33.5443146Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5443250Z self._dist_train() 2022-11-23T03:11:33.5443488Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5443737Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5443968Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5444099Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5444290Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5444426Z raise self.failureException(msg) 2022-11-23T03:11:33.5444557Z AssertionError: False is not true 2022-11-23T03:11:33.5444563Z 2022-11-23T03:11:33.5444569Z 2022-11-23T03:11:33.5444575Z 2022-11-23T03:11:33.5444797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49814 2022-11-23T03:11:33.5445020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49815 2022-11-23T03:11:33.5445424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5445600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5446012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5446286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5446536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5446938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5447114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5447522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5447720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5447973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5448409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5448828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5449058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5449339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5450488Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5450594Z warnings.warn( 2022-11-23T03:11:33.5450801Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5450928Z Traceback (most recent call last): 2022-11-23T03:11:33.5451332Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5451450Z getattr(self, test_name)() 2022-11-23T03:11:33.5451849Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5451940Z fn() 2022-11-23T03:11:33.5452331Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5452449Z return func(*args, **kwargs) 2022-11-23T03:11:33.5452699Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5452800Z self._dist_train() 2022-11-23T03:11:33.5453033Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5453296Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5453524Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5453660Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5453854Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5453993Z raise self.failureException(msg) 2022-11-23T03:11:33.5454108Z AssertionError: False is not true 2022-11-23T03:11:33.5454238Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5454339Z dist init r=1, world=2 2022-11-23T03:11:33.5455493Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5455650Z warnings.warn( 2022-11-23T03:11:33.5455855Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5455981Z Traceback (most recent call last): 2022-11-23T03:11:33.5456379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5456496Z getattr(self, test_name)() 2022-11-23T03:11:33.5456884Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5456975Z fn() 2022-11-23T03:11:33.5457360Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5457477Z return func(*args, **kwargs) 2022-11-23T03:11:33.5457727Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5457831Z self._dist_train() 2022-11-23T03:11:33.5458069Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5458320Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5458589Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5458727Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5458906Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5459044Z raise self.failureException(msg) 2022-11-23T03:11:33.5459173Z AssertionError: False is not true 2022-11-23T03:11:33.5459304Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5459408Z dist init r=0, world=2 2022-11-23T03:11:33.5459613Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5459713Z ERROR (3.731s) 2022-11-23T03:11:33.5460154Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 19 2022-11-23T03:11:33.5460281Z Traceback (most recent call last): 2022-11-23T03:11:33.5460669Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5460787Z self._join_processes(fn) 2022-11-23T03:11:33.5461198Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5461341Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5461756Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5461872Z raise RuntimeError(error) 2022-11-23T03:11:33.5462055Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5462170Z Traceback (most recent call last): 2022-11-23T03:11:33.5462567Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5462678Z getattr(self, test_name)() 2022-11-23T03:11:33.5463056Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5463147Z fn() 2022-11-23T03:11:33.5463531Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5463650Z return func(*args, **kwargs) 2022-11-23T03:11:33.5463898Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5464002Z self._dist_train() 2022-11-23T03:11:33.5464238Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5464490Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5464763Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5464896Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5465088Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5465229Z raise self.failureException(msg) 2022-11-23T03:11:33.5465359Z AssertionError: False is not true 2022-11-23T03:11:33.5465367Z 2022-11-23T03:11:33.5465374Z 2022-11-23T03:11:33.5465380Z 2022-11-23T03:11:33.5465601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49945 2022-11-23T03:11:33.5465821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49946 2022-11-23T03:11:33.5466228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5466388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5466796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5466990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5467239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5467685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5467862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5468275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5468466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5468719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5469159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5469591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5469822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5470059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5471199Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5471309Z warnings.warn( 2022-11-23T03:11:33.5471515Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5471645Z Traceback (most recent call last): 2022-11-23T03:11:33.5472044Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5472165Z getattr(self, test_name)() 2022-11-23T03:11:33.5472562Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5472648Z fn() 2022-11-23T03:11:33.5473033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5473154Z return func(*args, **kwargs) 2022-11-23T03:11:33.5473392Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5473494Z self._dist_train() 2022-11-23T03:11:33.5473732Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5474044Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5474272Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5474408Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5474604Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5474743Z raise self.failureException(msg) 2022-11-23T03:11:33.5474872Z AssertionError: False is not true 2022-11-23T03:11:33.5475002Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5475106Z dist init r=1, world=2 2022-11-23T03:11:33.5476245Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5476356Z warnings.warn( 2022-11-23T03:11:33.5476556Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5476681Z Traceback (most recent call last): 2022-11-23T03:11:33.5477121Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5477239Z getattr(self, test_name)() 2022-11-23T03:11:33.5477630Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5477718Z fn() 2022-11-23T03:11:33.5478095Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5478215Z return func(*args, **kwargs) 2022-11-23T03:11:33.5478466Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5478571Z self._dist_train() 2022-11-23T03:11:33.5478811Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5479062Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5479290Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5479424Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5479616Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5479753Z raise self.failureException(msg) 2022-11-23T03:11:33.5479884Z AssertionError: False is not true 2022-11-23T03:11:33.5480014Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5480117Z dist init r=0, world=2 2022-11-23T03:11:33.5480392Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5480490Z ERROR (3.833s) 2022-11-23T03:11:33.5480924Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 18 2022-11-23T03:11:33.5481039Z Traceback (most recent call last): 2022-11-23T03:11:33.5481432Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5481550Z self._join_processes(fn) 2022-11-23T03:11:33.5481963Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5482106Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5482524Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5482650Z raise RuntimeError(error) 2022-11-23T03:11:33.5482841Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5482974Z Traceback (most recent call last): 2022-11-23T03:11:33.5483425Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5483543Z getattr(self, test_name)() 2022-11-23T03:11:33.5483928Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5484019Z fn() 2022-11-23T03:11:33.5484413Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5484530Z return func(*args, **kwargs) 2022-11-23T03:11:33.5484783Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5484885Z self._dist_train() 2022-11-23T03:11:33.5485109Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5485358Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5485583Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5485720Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5485910Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5486048Z raise self.failureException(msg) 2022-11-23T03:11:33.5486353Z AssertionError: False is not true 2022-11-23T03:11:33.5486361Z 2022-11-23T03:11:33.5486367Z 2022-11-23T03:11:33.5486373Z 2022-11-23T03:11:33.5486597Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50076 2022-11-23T03:11:33.5486815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50077 2022-11-23T03:11:33.5487224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5487399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5487813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5488012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5488265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5488672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5496217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5496734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5496921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5497170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5497611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5498042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5498268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5498493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5499642Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5499739Z warnings.warn( 2022-11-23T03:11:33.5499936Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5500198Z Traceback (most recent call last): 2022-11-23T03:11:33.5500595Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5500711Z getattr(self, test_name)() 2022-11-23T03:11:33.5501107Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5501193Z fn() 2022-11-23T03:11:33.5501579Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5501699Z return func(*args, **kwargs) 2022-11-23T03:11:33.5501954Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5502057Z self._dist_train() 2022-11-23T03:11:33.5502290Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5502547Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5502777Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5502914Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5503094Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5503299Z raise self.failureException(msg) 2022-11-23T03:11:33.5503434Z AssertionError: False is not true 2022-11-23T03:11:33.5503568Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5504710Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5504822Z warnings.warn( 2022-11-23T03:11:33.5505023Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5505148Z Traceback (most recent call last): 2022-11-23T03:11:33.5505545Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5505660Z getattr(self, test_name)() 2022-11-23T03:11:33.5506047Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5506131Z fn() 2022-11-23T03:11:33.5506515Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5506631Z return func(*args, **kwargs) 2022-11-23T03:11:33.5506883Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5506985Z self._dist_train() 2022-11-23T03:11:33.5507226Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5507479Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5507710Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5507834Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5508090Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5508193Z raise self.failureException(msg) 2022-11-23T03:11:33.5508284Z AssertionError: False is not true 2022-11-23T03:11:33.5508381Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5508457Z dist init r=0, world=2 2022-11-23T03:11:33.5508532Z dist init r=1, world=2 2022-11-23T03:11:33.5508677Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5508747Z ERROR (3.731s) 2022-11-23T03:11:33.5509051Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 17 2022-11-23T03:11:33.5509178Z Traceback (most recent call last): 2022-11-23T03:11:33.5509456Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5509542Z self._join_processes(fn) 2022-11-23T03:11:33.5509836Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5509938Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5510235Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5510310Z raise RuntimeError(error) 2022-11-23T03:11:33.5510446Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5510538Z Traceback (most recent call last): 2022-11-23T03:11:33.5510817Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5510901Z getattr(self, test_name)() 2022-11-23T03:11:33.5511177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5511241Z fn() 2022-11-23T03:11:33.5511546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5511632Z return func(*args, **kwargs) 2022-11-23T03:11:33.5511812Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5511886Z self._dist_train() 2022-11-23T03:11:33.5512055Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5512231Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5512390Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5512485Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5512620Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5512717Z raise self.failureException(msg) 2022-11-23T03:11:33.5512800Z AssertionError: False is not true 2022-11-23T03:11:33.5512817Z 2022-11-23T03:11:33.5512822Z 2022-11-23T03:11:33.5512826Z 2022-11-23T03:11:33.5512983Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50207 2022-11-23T03:11:33.5513129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50208 2022-11-23T03:11:33.5513411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5513537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5513826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5513965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5514142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5514425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5514556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5514844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5514978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5515157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5515465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5515812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5515979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5516142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5516950Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5517025Z warnings.warn( 2022-11-23T03:11:33.5517169Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5517261Z Traceback (most recent call last): 2022-11-23T03:11:33.5517541Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5517626Z getattr(self, test_name)() 2022-11-23T03:11:33.5517959Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5518026Z fn() 2022-11-23T03:11:33.5518293Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5518380Z return func(*args, **kwargs) 2022-11-23T03:11:33.5518560Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5518635Z self._dist_train() 2022-11-23T03:11:33.5518802Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5518982Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5519143Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5519237Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5519370Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5519471Z raise self.failureException(msg) 2022-11-23T03:11:33.5519561Z AssertionError: False is not true 2022-11-23T03:11:33.5519654Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5519726Z dist init r=1, world=2 2022-11-23T03:11:33.5520961Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5521045Z warnings.warn( 2022-11-23T03:11:33.5521202Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5521303Z Traceback (most recent call last): 2022-11-23T03:11:33.5521614Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5521703Z getattr(self, test_name)() 2022-11-23T03:11:33.5521991Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5522056Z fn() 2022-11-23T03:11:33.5522359Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5522449Z return func(*args, **kwargs) 2022-11-23T03:11:33.5522644Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5522774Z self._dist_train() 2022-11-23T03:11:33.5522958Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5523152Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5523327Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5523432Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5523578Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5523682Z raise self.failureException(msg) 2022-11-23T03:11:33.5523781Z AssertionError: False is not true 2022-11-23T03:11:33.5523882Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5523963Z dist init r=0, world=2 2022-11-23T03:11:33.5524120Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5524184Z ERROR (3.732s) 2022-11-23T03:11:33.5524516Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 16 2022-11-23T03:11:33.5524615Z Traceback (most recent call last): 2022-11-23T03:11:33.5524919Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5525011Z self._join_processes(fn) 2022-11-23T03:11:33.5525368Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5525478Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5525794Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5525885Z raise RuntimeError(error) 2022-11-23T03:11:33.5526027Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5526122Z Traceback (most recent call last): 2022-11-23T03:11:33.5526417Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5526504Z getattr(self, test_name)() 2022-11-23T03:11:33.5526796Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5526861Z fn() 2022-11-23T03:11:33.5527158Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5527247Z return func(*args, **kwargs) 2022-11-23T03:11:33.5527439Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5527508Z self._dist_train() 2022-11-23T03:11:33.5527689Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5527882Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5528061Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5528166Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5528312Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5528418Z raise self.failureException(msg) 2022-11-23T03:11:33.5528517Z AssertionError: False is not true 2022-11-23T03:11:33.5528523Z 2022-11-23T03:11:33.5528530Z 2022-11-23T03:11:33.5528534Z 2022-11-23T03:11:33.5528708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50338 2022-11-23T03:11:33.5528876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50339 2022-11-23T03:11:33.5529179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5529313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5529624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5529811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5530001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5530307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5530439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5530749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5530891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5531080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5531516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5531841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5532006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5532290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5533136Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5533215Z warnings.warn( 2022-11-23T03:11:33.5533363Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5533459Z Traceback (most recent call last): 2022-11-23T03:11:33.5533745Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5533831Z getattr(self, test_name)() 2022-11-23T03:11:33.5534111Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5534175Z fn() 2022-11-23T03:11:33.5534450Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5534533Z return func(*args, **kwargs) 2022-11-23T03:11:33.5534708Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5534779Z self._dist_train() 2022-11-23T03:11:33.5534942Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5535119Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5535280Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5535373Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5535509Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5535612Z raise self.failureException(msg) 2022-11-23T03:11:33.5535695Z AssertionError: False is not true 2022-11-23T03:11:33.5535791Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5535867Z dist init r=0, world=2 2022-11-23T03:11:33.5536678Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5536795Z warnings.warn( 2022-11-23T03:11:33.5536940Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5537037Z Traceback (most recent call last): 2022-11-23T03:11:33.5537323Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5537410Z getattr(self, test_name)() 2022-11-23T03:11:33.5537688Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5537748Z fn() 2022-11-23T03:11:33.5538020Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5538105Z return func(*args, **kwargs) 2022-11-23T03:11:33.5538285Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5538361Z self._dist_train() 2022-11-23T03:11:33.5538529Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5538710Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5538866Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5539003Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5539129Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5539226Z raise self.failureException(msg) 2022-11-23T03:11:33.5539320Z AssertionError: False is not true 2022-11-23T03:11:33.5539415Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5539489Z dist init r=1, world=2 2022-11-23T03:11:33.5539634Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5539703Z ERROR (3.630s) 2022-11-23T03:11:33.5540007Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 15 2022-11-23T03:11:33.5540097Z Traceback (most recent call last): 2022-11-23T03:11:33.5540374Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5540456Z self._join_processes(fn) 2022-11-23T03:11:33.5540746Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5540850Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5541150Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5541236Z raise RuntimeError(error) 2022-11-23T03:11:33.5541370Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5541451Z Traceback (most recent call last): 2022-11-23T03:11:33.5541730Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5541817Z getattr(self, test_name)() 2022-11-23T03:11:33.5542092Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5542159Z fn() 2022-11-23T03:11:33.5542440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5542524Z return func(*args, **kwargs) 2022-11-23T03:11:33.5542697Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5542769Z self._dist_train() 2022-11-23T03:11:33.5542934Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5543114Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5543273Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5543408Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5543542Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5543642Z raise self.failureException(msg) 2022-11-23T03:11:33.5543737Z AssertionError: False is not true 2022-11-23T03:11:33.5543742Z 2022-11-23T03:11:33.5543749Z 2022-11-23T03:11:33.5543753Z 2022-11-23T03:11:33.5543911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50469 2022-11-23T03:11:33.5544070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50470 2022-11-23T03:11:33.5544357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5544480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5544759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5544898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5545075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5545357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5545521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5545812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5545947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5546130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5546434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5546736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5546903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5547062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5547866Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5547940Z warnings.warn( 2022-11-23T03:11:33.5548083Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5548170Z Traceback (most recent call last): 2022-11-23T03:11:33.5548623Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5548707Z getattr(self, test_name)() 2022-11-23T03:11:33.5549005Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5549072Z fn() 2022-11-23T03:11:33.5549362Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5549452Z return func(*args, **kwargs) 2022-11-23T03:11:33.5549639Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5549706Z self._dist_train() 2022-11-23T03:11:33.5549883Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5550075Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5550312Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5550417Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5550560Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5550671Z raise self.failureException(msg) 2022-11-23T03:11:33.5550768Z AssertionError: False is not true 2022-11-23T03:11:33.5550864Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5550940Z dist init r=0, world=2 2022-11-23T03:11:33.5551784Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5551865Z warnings.warn( 2022-11-23T03:11:33.5552018Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5552117Z Traceback (most recent call last): 2022-11-23T03:11:33.5552452Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5552543Z getattr(self, test_name)() 2022-11-23T03:11:33.5552836Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5552907Z fn() 2022-11-23T03:11:33.5553184Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5553274Z return func(*args, **kwargs) 2022-11-23T03:11:33.5553465Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5553544Z self._dist_train() 2022-11-23T03:11:33.5553722Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5553912Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5554086Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5554187Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5554329Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5554430Z raise self.failureException(msg) 2022-11-23T03:11:33.5554526Z AssertionError: False is not true 2022-11-23T03:11:33.5554621Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5554700Z dist init r=1, world=2 2022-11-23T03:11:33.5554852Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5554930Z ERROR (3.832s) 2022-11-23T03:11:33.5555255Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 14 2022-11-23T03:11:33.5555343Z Traceback (most recent call last): 2022-11-23T03:11:33.5555637Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5555728Z self._join_processes(fn) 2022-11-23T03:11:33.5556039Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5556145Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5556450Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5556543Z raise RuntimeError(error) 2022-11-23T03:11:33.5556682Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5556777Z Traceback (most recent call last): 2022-11-23T03:11:33.5557066Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5557193Z getattr(self, test_name)() 2022-11-23T03:11:33.5557482Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5557547Z fn() 2022-11-23T03:11:33.5557836Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5557920Z return func(*args, **kwargs) 2022-11-23T03:11:33.5558109Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5558186Z self._dist_train() 2022-11-23T03:11:33.5558364Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5558542Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5558708Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5558812Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5558957Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5559061Z raise self.failureException(msg) 2022-11-23T03:11:33.5559159Z AssertionError: False is not true 2022-11-23T03:11:33.5559164Z 2022-11-23T03:11:33.5559169Z 2022-11-23T03:11:33.5559210Z 2022-11-23T03:11:33.5559380Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50600 2022-11-23T03:11:33.5559545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50601 2022-11-23T03:11:33.5559848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5559977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5560368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5560680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5560874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5561179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5561315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5561633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5561781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5561974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5562308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5562640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5562820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5562999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5563876Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5563958Z warnings.warn( 2022-11-23T03:11:33.5564116Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5564205Z Traceback (most recent call last): 2022-11-23T03:11:33.5564560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5564651Z getattr(self, test_name)() 2022-11-23T03:11:33.5564948Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5565018Z fn() 2022-11-23T03:11:33.5565317Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5565406Z return func(*args, **kwargs) 2022-11-23T03:11:33.5565598Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5565677Z self._dist_train() 2022-11-23T03:11:33.5565859Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5566052Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5566228Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5566334Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5566480Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5566589Z raise self.failureException(msg) 2022-11-23T03:11:33.5566730Z AssertionError: False is not true 2022-11-23T03:11:33.5566830Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5566899Z dist init r=1, world=2 2022-11-23T03:11:33.5567775Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5567864Z warnings.warn( 2022-11-23T03:11:33.5568023Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5568124Z Traceback (most recent call last): 2022-11-23T03:11:33.5568435Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5568527Z getattr(self, test_name)() 2022-11-23T03:11:33.5568822Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5568891Z fn() 2022-11-23T03:11:33.5569185Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5569277Z return func(*args, **kwargs) 2022-11-23T03:11:33.5569468Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5569548Z self._dist_train() 2022-11-23T03:11:33.5569735Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5569932Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5570104Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5570211Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5570358Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5570467Z raise self.failureException(msg) 2022-11-23T03:11:33.5570557Z AssertionError: False is not true 2022-11-23T03:11:33.5570661Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5570740Z dist init r=0, world=2 2022-11-23T03:11:33.5570895Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5570966Z ERROR (3.631s) 2022-11-23T03:11:33.5571300Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 13 2022-11-23T03:11:33.5571457Z Traceback (most recent call last): 2022-11-23T03:11:33.5571762Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5571849Z self._join_processes(fn) 2022-11-23T03:11:33.5572162Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5572269Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5572588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5572679Z raise RuntimeError(error) 2022-11-23T03:11:33.5572822Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5572923Z Traceback (most recent call last): 2022-11-23T03:11:33.5573219Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5573300Z getattr(self, test_name)() 2022-11-23T03:11:33.5573598Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5573666Z fn() 2022-11-23T03:11:33.5573998Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5574090Z return func(*args, **kwargs) 2022-11-23T03:11:33.5574282Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5574361Z self._dist_train() 2022-11-23T03:11:33.5574547Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5574738Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5574909Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5575011Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5575156Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5575260Z raise self.failureException(msg) 2022-11-23T03:11:33.5575355Z AssertionError: False is not true 2022-11-23T03:11:33.5575361Z 2022-11-23T03:11:33.5575366Z 2022-11-23T03:11:33.5575370Z 2022-11-23T03:11:33.5575542Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50731 2022-11-23T03:11:33.5575710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50732 2022-11-23T03:11:33.5576017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5576152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5576467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5576611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5576797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5577103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5577237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5577555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5577701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5577895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5578226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5578557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5578773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5578948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5579831Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5579918Z warnings.warn( 2022-11-23T03:11:33.5580075Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5580173Z Traceback (most recent call last): 2022-11-23T03:11:33.5580478Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5580570Z getattr(self, test_name)() 2022-11-23T03:11:33.5580870Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5580940Z fn() 2022-11-23T03:11:33.5581385Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5581478Z return func(*args, **kwargs) 2022-11-23T03:11:33.5581668Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5581745Z self._dist_train() 2022-11-23T03:11:33.5581920Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5582099Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5582373Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5582470Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5582606Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5582708Z raise self.failureException(msg) 2022-11-23T03:11:33.5582802Z AssertionError: False is not true 2022-11-23T03:11:33.5582893Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5582969Z dist init r=0, world=2 2022-11-23T03:11:33.5583775Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5583850Z warnings.warn( 2022-11-23T03:11:33.5583992Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5584084Z Traceback (most recent call last): 2022-11-23T03:11:33.5584362Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5584448Z getattr(self, test_name)() 2022-11-23T03:11:33.5584725Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5584789Z fn() 2022-11-23T03:11:33.5585063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5585148Z return func(*args, **kwargs) 2022-11-23T03:11:33.5585322Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5585385Z self._dist_train() 2022-11-23T03:11:33.5585548Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5585765Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5585927Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5586024Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5586161Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5586264Z raise self.failureException(msg) 2022-11-23T03:11:33.5586357Z AssertionError: False is not true 2022-11-23T03:11:33.5586450Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5586521Z dist init r=1, world=2 2022-11-23T03:11:33.5586663Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5586729Z ERROR (3.632s) 2022-11-23T03:11:33.5587031Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 12 2022-11-23T03:11:33.5587126Z Traceback (most recent call last): 2022-11-23T03:11:33.5587408Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5587494Z self._join_processes(fn) 2022-11-23T03:11:33.5587807Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5587913Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5588206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5588295Z raise RuntimeError(error) 2022-11-23T03:11:33.5588431Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5588526Z Traceback (most recent call last): 2022-11-23T03:11:33.5588800Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5588884Z getattr(self, test_name)() 2022-11-23T03:11:33.5589157Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5589224Z fn() 2022-11-23T03:11:33.5589496Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5589579Z return func(*args, **kwargs) 2022-11-23T03:11:33.5589758Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5589834Z self._dist_train() 2022-11-23T03:11:33.5590004Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5590182Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5590337Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5590422Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5590558Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5590657Z raise self.failureException(msg) 2022-11-23T03:11:33.5590749Z AssertionError: False is not true 2022-11-23T03:11:33.5590754Z 2022-11-23T03:11:33.5590758Z 2022-11-23T03:11:33.5590762Z 2022-11-23T03:11:33.5590923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50862 2022-11-23T03:11:33.5591082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50863 2022-11-23T03:11:33.5591366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5591487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5591777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5591909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5592119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5592400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5592523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5592808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5592944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5593124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5593432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5593735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5593905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5594067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5594905Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5594982Z warnings.warn( 2022-11-23T03:11:33.5595126Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5595217Z Traceback (most recent call last): 2022-11-23T03:11:33.5595497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5595572Z getattr(self, test_name)() 2022-11-23T03:11:33.5595851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5595915Z fn() 2022-11-23T03:11:33.5596191Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5596280Z return func(*args, **kwargs) 2022-11-23T03:11:33.5596459Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5596530Z self._dist_train() 2022-11-23T03:11:33.5596699Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5596874Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5597030Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5597131Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5597269Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5597368Z raise self.failureException(msg) 2022-11-23T03:11:33.5597460Z AssertionError: False is not true 2022-11-23T03:11:33.5597553Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5597630Z dist init r=1, world=2 2022-11-23T03:11:33.5598439Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5598515Z warnings.warn( 2022-11-23T03:11:33.5598702Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5598785Z Traceback (most recent call last): 2022-11-23T03:11:33.5599065Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5599146Z getattr(self, test_name)() 2022-11-23T03:11:33.5599423Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5599487Z fn() 2022-11-23T03:11:33.5599910Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5599994Z return func(*args, **kwargs) 2022-11-23T03:11:33.5600227Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5600460Z self._dist_train() 2022-11-23T03:11:33.5600638Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5600829Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5600994Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5601097Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5601289Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5601397Z raise self.failureException(msg) 2022-11-23T03:11:33.5601494Z AssertionError: False is not true 2022-11-23T03:11:33.5601595Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5601661Z dist init r=0, world=2 2022-11-23T03:11:33.5601814Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5601886Z ERROR (3.631s) 2022-11-23T03:11:33.5602215Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 11 2022-11-23T03:11:33.5602312Z Traceback (most recent call last): 2022-11-23T03:11:33.5602608Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5602695Z self._join_processes(fn) 2022-11-23T03:11:33.5603003Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5603108Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5603415Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5603500Z raise RuntimeError(error) 2022-11-23T03:11:33.5603639Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5603735Z Traceback (most recent call last): 2022-11-23T03:11:33.5604026Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5604113Z getattr(self, test_name)() 2022-11-23T03:11:33.5604404Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5604472Z fn() 2022-11-23T03:11:33.5604747Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5604842Z return func(*args, **kwargs) 2022-11-23T03:11:33.5605027Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5605101Z self._dist_train() 2022-11-23T03:11:33.5605281Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5605471Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5605638Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5605735Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5605919Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5606018Z raise self.failureException(msg) 2022-11-23T03:11:33.5606113Z AssertionError: False is not true 2022-11-23T03:11:33.5606119Z 2022-11-23T03:11:33.5606124Z 2022-11-23T03:11:33.5606128Z 2022-11-23T03:11:33.5606289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50993 2022-11-23T03:11:33.5606452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50994 2022-11-23T03:11:33.5606759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5606889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5607192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5607333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5607524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5607822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5607953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5608287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5608431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5608618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5608944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5609266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5609443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5609616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5610471Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5610551Z warnings.warn( 2022-11-23T03:11:33.5610705Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5610801Z Traceback (most recent call last): 2022-11-23T03:11:33.5611098Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5611189Z getattr(self, test_name)() 2022-11-23T03:11:33.5611484Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5611551Z fn() 2022-11-23T03:11:33.5611848Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5611938Z return func(*args, **kwargs) 2022-11-23T03:11:33.5612235Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5612308Z self._dist_train() 2022-11-23T03:11:33.5612480Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5612658Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5612816Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5612937Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5613072Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5613252Z raise self.failureException(msg) 2022-11-23T03:11:33.5613342Z AssertionError: False is not true 2022-11-23T03:11:33.5613433Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5613509Z dist init r=1, world=2 2022-11-23T03:11:33.5614319Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5614392Z warnings.warn( 2022-11-23T03:11:33.5614537Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5614627Z Traceback (most recent call last): 2022-11-23T03:11:33.5614904Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5614984Z getattr(self, test_name)() 2022-11-23T03:11:33.5615289Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5615356Z fn() 2022-11-23T03:11:33.5615635Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5615720Z return func(*args, **kwargs) 2022-11-23T03:11:33.5615898Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5615973Z self._dist_train() 2022-11-23T03:11:33.5616137Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5616307Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5616465Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5616560Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5616694Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5616792Z raise self.failureException(msg) 2022-11-23T03:11:33.5616883Z AssertionError: False is not true 2022-11-23T03:11:33.5616975Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5617048Z dist init r=0, world=2 2022-11-23T03:11:33.5617193Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5617258Z ERROR (3.835s) 2022-11-23T03:11:33.5617562Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 10 2022-11-23T03:11:33.5617652Z Traceback (most recent call last): 2022-11-23T03:11:33.5617932Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5618015Z self._join_processes(fn) 2022-11-23T03:11:33.5618306Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5618407Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5618687Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5618775Z raise RuntimeError(error) 2022-11-23T03:11:33.5618907Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5618998Z Traceback (most recent call last): 2022-11-23T03:11:33.5619270Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5619353Z getattr(self, test_name)() 2022-11-23T03:11:33.5619668Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5619731Z fn() 2022-11-23T03:11:33.5620000Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5620084Z return func(*args, **kwargs) 2022-11-23T03:11:33.5620259Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5620333Z self._dist_train() 2022-11-23T03:11:33.5620499Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5620681Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5620837Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5620933Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5621071Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5621159Z raise self.failureException(msg) 2022-11-23T03:11:33.5621253Z AssertionError: False is not true 2022-11-23T03:11:33.5621258Z 2022-11-23T03:11:33.5621263Z 2022-11-23T03:11:33.5621276Z 2022-11-23T03:11:33.5621476Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51124 2022-11-23T03:11:33.5621635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51125 2022-11-23T03:11:33.5621922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5622048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5622340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5622478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5622654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5622937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5623060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5623346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5623482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5623660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5623966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5624266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5624435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5624599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5625404Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5625479Z warnings.warn( 2022-11-23T03:11:33.5625621Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5625714Z Traceback (most recent call last): 2022-11-23T03:11:33.5625992Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5626106Z getattr(self, test_name)() 2022-11-23T03:11:33.5626382Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5626446Z fn() 2022-11-23T03:11:33.5626713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5626795Z return func(*args, **kwargs) 2022-11-23T03:11:33.5626971Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5627043Z self._dist_train() 2022-11-23T03:11:33.5627211Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5627394Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5627554Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5627648Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5627782Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5627878Z raise self.failureException(msg) 2022-11-23T03:11:33.5627973Z AssertionError: False is not true 2022-11-23T03:11:33.5628104Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5628913Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5628986Z warnings.warn( 2022-11-23T03:11:33.5629130Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5629228Z Traceback (most recent call last): 2022-11-23T03:11:33.5629508Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5629590Z getattr(self, test_name)() 2022-11-23T03:11:33.5629867Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5629920Z fn() 2022-11-23T03:11:33.5630192Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5630276Z return func(*args, **kwargs) 2022-11-23T03:11:33.5630457Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5630533Z self._dist_train() 2022-11-23T03:11:33.5630699Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5630878Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5631039Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5631131Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5631265Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5631367Z raise self.failureException(msg) 2022-11-23T03:11:33.5631460Z AssertionError: False is not true 2022-11-23T03:11:33.5631554Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5631628Z dist init r=1, world=2 2022-11-23T03:11:33.5631699Z dist init r=0, world=2 2022-11-23T03:11:33.5631833Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5631904Z ERROR (3.832s) 2022-11-23T03:11:33.5632207Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 9 2022-11-23T03:11:33.5632298Z Traceback (most recent call last): 2022-11-23T03:11:33.5632612Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5632696Z self._join_processes(fn) 2022-11-23T03:11:33.5632984Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5633087Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5633380Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5633462Z raise RuntimeError(error) 2022-11-23T03:11:33.5633591Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5633682Z Traceback (most recent call last): 2022-11-23T03:11:33.5633954Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5634037Z getattr(self, test_name)() 2022-11-23T03:11:33.5634313Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5634380Z fn() 2022-11-23T03:11:33.5634653Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5634738Z return func(*args, **kwargs) 2022-11-23T03:11:33.5634940Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5635015Z self._dist_train() 2022-11-23T03:11:33.5635187Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5635366Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5635526Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5635621Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5635754Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5635854Z raise self.failureException(msg) 2022-11-23T03:11:33.5635946Z AssertionError: False is not true 2022-11-23T03:11:33.5635951Z 2022-11-23T03:11:33.5635955Z 2022-11-23T03:11:33.5635960Z 2022-11-23T03:11:33.5636114Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51255 2022-11-23T03:11:33.5636276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51256 2022-11-23T03:11:33.5636561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5636683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5636973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5637111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5637286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5637570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5637695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5637987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5638123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5638303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5638611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5638917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5639122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5639285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5640406Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5640658Z warnings.warn( 2022-11-23T03:11:33.5640817Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5640906Z Traceback (most recent call last): 2022-11-23T03:11:33.5641216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5641310Z getattr(self, test_name)() 2022-11-23T03:11:33.5641613Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5641683Z fn() 2022-11-23T03:11:33.5642027Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5642119Z return func(*args, **kwargs) 2022-11-23T03:11:33.5642311Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5642388Z self._dist_train() 2022-11-23T03:11:33.5642569Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5642766Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5642942Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5643051Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5643201Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5643310Z raise self.failureException(msg) 2022-11-23T03:11:33.5643412Z AssertionError: False is not true 2022-11-23T03:11:33.5643516Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5644393Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5644474Z warnings.warn( 2022-11-23T03:11:33.5644632Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5644723Z Traceback (most recent call last): 2022-11-23T03:11:33.5645030Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5645126Z getattr(self, test_name)() 2022-11-23T03:11:33.5645434Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5645508Z fn() 2022-11-23T03:11:33.5645809Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5645898Z return func(*args, **kwargs) 2022-11-23T03:11:33.5646087Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5646162Z self._dist_train() 2022-11-23T03:11:33.5646341Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5646536Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5646752Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5646858Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5647012Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5647125Z raise self.failureException(msg) 2022-11-23T03:11:33.5647229Z AssertionError: False is not true 2022-11-23T03:11:33.5647320Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5647403Z dist init r=0, world=2 2022-11-23T03:11:33.5647482Z dist init r=1, world=2 2022-11-23T03:11:33.5647639Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5647717Z ERROR (3.631s) 2022-11-23T03:11:33.5648054Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 8 2022-11-23T03:11:33.5648155Z Traceback (most recent call last): 2022-11-23T03:11:33.5648457Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5648546Z self._join_processes(fn) 2022-11-23T03:11:33.5648860Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5649010Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5649333Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5649424Z raise RuntimeError(error) 2022-11-23T03:11:33.5649568Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5649671Z Traceback (most recent call last): 2022-11-23T03:11:33.5649974Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5650064Z getattr(self, test_name)() 2022-11-23T03:11:33.5650351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5650422Z fn() 2022-11-23T03:11:33.5650718Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5650810Z return func(*args, **kwargs) 2022-11-23T03:11:33.5651007Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5651195Z self._dist_train() 2022-11-23T03:11:33.5651375Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5651564Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5651731Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5651828Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5652082Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5652181Z raise self.failureException(msg) 2022-11-23T03:11:33.5652270Z AssertionError: False is not true 2022-11-23T03:11:33.5652276Z 2022-11-23T03:11:33.5652280Z 2022-11-23T03:11:33.5652284Z 2022-11-23T03:11:33.5652440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51386 2022-11-23T03:11:33.5652599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51387 2022-11-23T03:11:33.5652883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5653010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5653304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5653443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5653623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5653935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5654063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5654356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5654493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5654674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5654984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5655285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5655454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5655617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5656456Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5656533Z warnings.warn( 2022-11-23T03:11:33.5656681Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5656776Z Traceback (most recent call last): 2022-11-23T03:11:33.5657059Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5657147Z getattr(self, test_name)() 2022-11-23T03:11:33.5657427Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5657489Z fn() 2022-11-23T03:11:33.5657768Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5657856Z return func(*args, **kwargs) 2022-11-23T03:11:33.5658032Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5658103Z self._dist_train() 2022-11-23T03:11:33.5658273Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5658443Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5658612Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5658711Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5658845Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5658946Z raise self.failureException(msg) 2022-11-23T03:11:33.5659038Z AssertionError: False is not true 2022-11-23T03:11:33.5659128Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5659207Z dist init r=1, world=2 2022-11-23T03:11:33.5660010Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5660088Z warnings.warn( 2022-11-23T03:11:33.5660233Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5660356Z Traceback (most recent call last): 2022-11-23T03:11:33.5660634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5660714Z getattr(self, test_name)() 2022-11-23T03:11:33.5660991Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5661051Z fn() 2022-11-23T03:11:33.5661328Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5661414Z return func(*args, **kwargs) 2022-11-23T03:11:33.5661588Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5661651Z self._dist_train() 2022-11-23T03:11:33.5661826Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5662007Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5662168Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5662262Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5662398Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5662534Z raise self.failureException(msg) 2022-11-23T03:11:33.5662630Z AssertionError: False is not true 2022-11-23T03:11:33.5662725Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5662802Z dist init r=0, world=2 2022-11-23T03:11:33.5662944Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5663016Z ERROR (3.732s) 2022-11-23T03:11:33.5663318Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 7 2022-11-23T03:11:33.5663406Z Traceback (most recent call last): 2022-11-23T03:11:33.5663683Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5663765Z self._join_processes(fn) 2022-11-23T03:11:33.5664040Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5664148Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5664442Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5664528Z raise RuntimeError(error) 2022-11-23T03:11:33.5664658Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5664751Z Traceback (most recent call last): 2022-11-23T03:11:33.5665026Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5665110Z getattr(self, test_name)() 2022-11-23T03:11:33.5665379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5665444Z fn() 2022-11-23T03:11:33.5665715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5665797Z return func(*args, **kwargs) 2022-11-23T03:11:33.5665976Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5666045Z self._dist_train() 2022-11-23T03:11:33.5666211Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5666392Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5666554Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5666649Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5666773Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5666910Z raise self.failureException(msg) 2022-11-23T03:11:33.5667005Z AssertionError: False is not true 2022-11-23T03:11:33.5667011Z 2022-11-23T03:11:33.5667015Z 2022-11-23T03:11:33.5667019Z 2022-11-23T03:11:33.5667177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51517 2022-11-23T03:11:33.5667333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51518 2022-11-23T03:11:33.5667617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5667740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5668029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5668165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5668341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5668625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5668744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5669075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5669213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5669390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5669700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5670006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5670171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5670338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5671141Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5671219Z warnings.warn( 2022-11-23T03:11:33.5671363Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5671454Z Traceback (most recent call last): 2022-11-23T03:11:33.5671732Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5671815Z getattr(self, test_name)() 2022-11-23T03:11:33.5672085Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5672146Z fn() 2022-11-23T03:11:33.5672424Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5672506Z return func(*args, **kwargs) 2022-11-23T03:11:33.5672678Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5672751Z self._dist_train() 2022-11-23T03:11:33.5672914Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5673094Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5673255Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5673406Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5673540Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5673639Z raise self.failureException(msg) 2022-11-23T03:11:33.5673732Z AssertionError: False is not true 2022-11-23T03:11:33.5673830Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5673913Z dist init r=1, world=2 2022-11-23T03:11:33.5674722Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5674801Z warnings.warn( 2022-11-23T03:11:33.5674956Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5675040Z Traceback (most recent call last): 2022-11-23T03:11:33.5675322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5675406Z getattr(self, test_name)() 2022-11-23T03:11:33.5675725Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5675789Z fn() 2022-11-23T03:11:33.5676063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5676147Z return func(*args, **kwargs) 2022-11-23T03:11:33.5676324Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5676395Z self._dist_train() 2022-11-23T03:11:33.5676559Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5676735Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5676894Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5676988Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5677121Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5677222Z raise self.failureException(msg) 2022-11-23T03:11:33.5677313Z AssertionError: False is not true 2022-11-23T03:11:33.5677409Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5677474Z dist init r=0, world=2 2022-11-23T03:11:33.5677617Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5677686Z ERROR (3.931s) 2022-11-23T03:11:33.5677989Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 6 2022-11-23T03:11:33.5678084Z Traceback (most recent call last): 2022-11-23T03:11:33.5678361Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5678447Z self._join_processes(fn) 2022-11-23T03:11:33.5678742Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5678845Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5679135Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5679219Z raise RuntimeError(error) 2022-11-23T03:11:33.5679352Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5679441Z Traceback (most recent call last): 2022-11-23T03:11:33.5679718Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5679799Z getattr(self, test_name)() 2022-11-23T03:11:33.5680073Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5680249Z fn() 2022-11-23T03:11:33.5680864Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5680961Z return func(*args, **kwargs) 2022-11-23T03:11:33.5681160Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5681241Z self._dist_train() 2022-11-23T03:11:33.5681427Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5681619Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5681792Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5681896Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5682040Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5682149Z raise self.failureException(msg) 2022-11-23T03:11:33.5682246Z AssertionError: False is not true 2022-11-23T03:11:33.5682251Z 2022-11-23T03:11:33.5682257Z 2022-11-23T03:11:33.5682261Z 2022-11-23T03:11:33.5682430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51648 2022-11-23T03:11:33.5682669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51649 2022-11-23T03:11:33.5682983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5683118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5683433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5683582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5683777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5684087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5684223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5684540Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5684675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5684870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5685203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5685535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5685714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5685893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5686768Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5686851Z warnings.warn( 2022-11-23T03:11:33.5687007Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5687105Z Traceback (most recent call last): 2022-11-23T03:11:33.5687407Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5687541Z getattr(self, test_name)() 2022-11-23T03:11:33.5687843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5687912Z fn() 2022-11-23T03:11:33.5688209Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5688308Z return func(*args, **kwargs) 2022-11-23T03:11:33.5688504Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5688585Z self._dist_train() 2022-11-23T03:11:33.5688767Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5688960Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5689132Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5689234Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5689372Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5689478Z raise self.failureException(msg) 2022-11-23T03:11:33.5689575Z AssertionError: False is not true 2022-11-23T03:11:33.5689673Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5689751Z dist init r=1, world=2 2022-11-23T03:11:33.5690661Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5690745Z warnings.warn( 2022-11-23T03:11:33.5690902Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5690999Z Traceback (most recent call last): 2022-11-23T03:11:33.5691403Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5691489Z getattr(self, test_name)() 2022-11-23T03:11:33.5691780Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5691846Z fn() 2022-11-23T03:11:33.5692133Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5692225Z return func(*args, **kwargs) 2022-11-23T03:11:33.5692415Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5692490Z self._dist_train() 2022-11-23T03:11:33.5692772Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5692941Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5693103Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5693204Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5693342Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5693445Z raise self.failureException(msg) 2022-11-23T03:11:33.5693542Z AssertionError: False is not true 2022-11-23T03:11:33.5693635Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5693705Z dist init r=0, world=2 2022-11-23T03:11:33.5693850Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5693919Z ERROR (3.732s) 2022-11-23T03:11:33.5694218Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 5 2022-11-23T03:11:33.5694306Z Traceback (most recent call last): 2022-11-23T03:11:33.5694581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5694701Z self._join_processes(fn) 2022-11-23T03:11:33.5694995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5695101Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5695400Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5695477Z raise RuntimeError(error) 2022-11-23T03:11:33.5695614Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5695709Z Traceback (most recent call last): 2022-11-23T03:11:33.5695987Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5696071Z getattr(self, test_name)() 2022-11-23T03:11:33.5696350Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5696420Z fn() 2022-11-23T03:11:33.5696691Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5696775Z return func(*args, **kwargs) 2022-11-23T03:11:33.5696987Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5697064Z self._dist_train() 2022-11-23T03:11:33.5697233Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5697418Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5697581Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5697684Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5697824Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5697931Z raise self.failureException(msg) 2022-11-23T03:11:33.5698012Z AssertionError: False is not true 2022-11-23T03:11:33.5698017Z 2022-11-23T03:11:33.5698036Z 2022-11-23T03:11:33.5698040Z 2022-11-23T03:11:33.5698189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51779 2022-11-23T03:11:33.5698349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51780 2022-11-23T03:11:33.5698789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5698914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5699206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5699344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5699525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5699811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5699930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5700221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5700355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5700532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5700841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5701149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5701316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5701520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5702334Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5702412Z warnings.warn( 2022-11-23T03:11:33.5702558Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5702654Z Traceback (most recent call last): 2022-11-23T03:11:33.5702934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5703019Z getattr(self, test_name)() 2022-11-23T03:11:33.5703303Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5703369Z fn() 2022-11-23T03:11:33.5703633Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5703751Z return func(*args, **kwargs) 2022-11-23T03:11:33.5703930Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5704001Z self._dist_train() 2022-11-23T03:11:33.5704169Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5704352Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5704513Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5704613Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5704756Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5704854Z raise self.failureException(msg) 2022-11-23T03:11:33.5704947Z AssertionError: False is not true 2022-11-23T03:11:33.5705042Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5705846Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5705922Z warnings.warn( 2022-11-23T03:11:33.5706063Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5706153Z Traceback (most recent call last): 2022-11-23T03:11:33.5706433Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5706520Z getattr(self, test_name)() 2022-11-23T03:11:33.5706798Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5706852Z fn() 2022-11-23T03:11:33.5707123Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5707206Z return func(*args, **kwargs) 2022-11-23T03:11:33.5707381Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5707456Z self._dist_train() 2022-11-23T03:11:33.5707623Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5707797Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5707956Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5708088Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5708222Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5708320Z raise self.failureException(msg) 2022-11-23T03:11:33.5708414Z AssertionError: False is not true 2022-11-23T03:11:33.5708510Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5708588Z dist init r=0, world=2 2022-11-23T03:11:33.5708666Z dist init r=1, world=2 2022-11-23T03:11:33.5708813Z Process 0 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5708873Z ERROR (3.730s) 2022-11-23T03:11:33.5709177Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 4 2022-11-23T03:11:33.5709273Z Traceback (most recent call last): 2022-11-23T03:11:33.5709548Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5709634Z self._join_processes(fn) 2022-11-23T03:11:33.5709921Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5710021Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5710346Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5710432Z raise RuntimeError(error) 2022-11-23T03:11:33.5710566Z RuntimeError: Process 0 exited with error code 10 and exception: 2022-11-23T03:11:33.5710657Z Traceback (most recent call last): 2022-11-23T03:11:33.5710933Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5711016Z getattr(self, test_name)() 2022-11-23T03:11:33.5711291Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5711361Z fn() 2022-11-23T03:11:33.5711637Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5711724Z return func(*args, **kwargs) 2022-11-23T03:11:33.5711890Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5711962Z self._dist_train() 2022-11-23T03:11:33.5712127Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5712309Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5712468Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5712564Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5712700Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5712796Z raise self.failureException(msg) 2022-11-23T03:11:33.5712890Z AssertionError: False is not true 2022-11-23T03:11:33.5712896Z 2022-11-23T03:11:33.5712901Z 2022-11-23T03:11:33.5712905Z 2022-11-23T03:11:33.5713063Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51910 2022-11-23T03:11:33.5713217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51911 2022-11-23T03:11:33.5713497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5713617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5713905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5714039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5714218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5714657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5714780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5715073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5715211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5715388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5715695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5715992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5716162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5716329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5717169Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5717246Z warnings.warn( 2022-11-23T03:11:33.5717389Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5717478Z Traceback (most recent call last): 2022-11-23T03:11:33.5717760Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5717844Z getattr(self, test_name)() 2022-11-23T03:11:33.5718124Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5718186Z fn() 2022-11-23T03:11:33.5718463Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5718548Z return func(*args, **kwargs) 2022-11-23T03:11:33.5718724Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5718797Z self._dist_train() 2022-11-23T03:11:33.5718969Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5719143Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5719299Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5719393Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5719528Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5719629Z raise self.failureException(msg) 2022-11-23T03:11:33.5719710Z AssertionError: False is not true 2022-11-23T03:11:33.5719803Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5719874Z dist init r=1, world=2 2022-11-23T03:11:33.5720933Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5721014Z warnings.warn( 2022-11-23T03:11:33.5721168Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5721264Z Traceback (most recent call last): 2022-11-23T03:11:33.5721616Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5721705Z getattr(self, test_name)() 2022-11-23T03:11:33.5722000Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5722069Z fn() 2022-11-23T03:11:33.5722364Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5722456Z return func(*args, **kwargs) 2022-11-23T03:11:33.5722647Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5722725Z self._dist_train() 2022-11-23T03:11:33.5722903Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5723092Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5723266Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5723368Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5723501Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5723610Z raise self.failureException(msg) 2022-11-23T03:11:33.5723768Z AssertionError: False is not true 2022-11-23T03:11:33.5723867Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5723942Z dist init r=0, world=2 2022-11-23T03:11:33.5724092Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5724163Z ERROR (3.832s) 2022-11-23T03:11:33.5724481Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 3 2022-11-23T03:11:33.5724577Z Traceback (most recent call last): 2022-11-23T03:11:33.5724869Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5724958Z self._join_processes(fn) 2022-11-23T03:11:33.5725263Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5725372Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5725685Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5725778Z raise RuntimeError(error) 2022-11-23T03:11:33.5725922Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5726007Z Traceback (most recent call last): 2022-11-23T03:11:33.5726304Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5726392Z getattr(self, test_name)() 2022-11-23T03:11:33.5726685Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5726753Z fn() 2022-11-23T03:11:33.5727041Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5727129Z return func(*args, **kwargs) 2022-11-23T03:11:33.5727317Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5727396Z self._dist_train() 2022-11-23T03:11:33.5727569Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5727756Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5727924Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5728024Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5728168Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5728272Z raise self.failureException(msg) 2022-11-23T03:11:33.5728412Z AssertionError: False is not true 2022-11-23T03:11:33.5728418Z 2022-11-23T03:11:33.5728423Z 2022-11-23T03:11:33.5728426Z 2022-11-23T03:11:33.5728591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52041 2022-11-23T03:11:33.5728754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52042 2022-11-23T03:11:33.5729056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5729176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5729485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5729628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5729814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5730111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5730244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5730545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5730723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5730910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5731236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5731553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5731727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5731903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5732825Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5732904Z warnings.warn( 2022-11-23T03:11:33.5733049Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5733143Z Traceback (most recent call last): 2022-11-23T03:11:33.5733426Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5733509Z getattr(self, test_name)() 2022-11-23T03:11:33.5733792Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5733858Z fn() 2022-11-23T03:11:33.5734129Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5734213Z return func(*args, **kwargs) 2022-11-23T03:11:33.5734382Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5734456Z self._dist_train() 2022-11-23T03:11:33.5734625Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5734802Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5734958Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5735054Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5739741Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5739956Z raise self.failureException(msg) 2022-11-23T03:11:33.5740053Z AssertionError: False is not true 2022-11-23T03:11:33.5740150Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5740220Z dist init r=0, world=2 2022-11-23T03:11:33.5741075Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5741156Z warnings.warn( 2022-11-23T03:11:33.5741303Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5741400Z Traceback (most recent call last): 2022-11-23T03:11:33.5741679Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5741765Z getattr(self, test_name)() 2022-11-23T03:11:33.5742051Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5742115Z fn() 2022-11-23T03:11:33.5742450Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5742538Z return func(*args, **kwargs) 2022-11-23T03:11:33.5742719Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5742792Z self._dist_train() 2022-11-23T03:11:33.5742960Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5743140Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5743301Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5743397Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5743531Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5743628Z raise self.failureException(msg) 2022-11-23T03:11:33.5743721Z AssertionError: False is not true 2022-11-23T03:11:33.5743814Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5743888Z dist init r=1, world=2 2022-11-23T03:11:33.5744022Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5744092Z ERROR (3.833s) 2022-11-23T03:11:33.5744402Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 2 2022-11-23T03:11:33.5744492Z Traceback (most recent call last): 2022-11-23T03:11:33.5744765Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5744848Z self._join_processes(fn) 2022-11-23T03:11:33.5745138Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5745239Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5745533Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5745617Z raise RuntimeError(error) 2022-11-23T03:11:33.5745752Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5745842Z Traceback (most recent call last): 2022-11-23T03:11:33.5746118Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5746199Z getattr(self, test_name)() 2022-11-23T03:11:33.5746473Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5746578Z fn() 2022-11-23T03:11:33.5746851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5746925Z return func(*args, **kwargs) 2022-11-23T03:11:33.5747100Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5747179Z self._dist_train() 2022-11-23T03:11:33.5747351Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5747532Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5747691Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5747788Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5747923Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5748022Z raise self.failureException(msg) 2022-11-23T03:11:33.5748113Z AssertionError: False is not true 2022-11-23T03:11:33.5748119Z 2022-11-23T03:11:33.5748124Z 2022-11-23T03:11:33.5748128Z 2022-11-23T03:11:33.5748283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52172 2022-11-23T03:11:33.5748437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52173 2022-11-23T03:11:33.5748754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5748879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5749172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5749306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5749484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5749768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5749893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5750183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5750321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5750489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5750795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5751091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5751254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5751417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5752228Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5752303Z warnings.warn( 2022-11-23T03:11:33.5752444Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5752536Z Traceback (most recent call last): 2022-11-23T03:11:33.5752815Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5752899Z getattr(self, test_name)() 2022-11-23T03:11:33.5753175Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5753277Z fn() 2022-11-23T03:11:33.5753550Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5753631Z return func(*args, **kwargs) 2022-11-23T03:11:33.5753811Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5753884Z self._dist_train() 2022-11-23T03:11:33.5754054Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5754233Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5754391Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5754486Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5754611Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5754710Z raise self.failureException(msg) 2022-11-23T03:11:33.5754802Z AssertionError: False is not true 2022-11-23T03:11:33.5754894Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5754966Z dist init r=1, world=2 2022-11-23T03:11:33.5755803Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5755877Z warnings.warn( 2022-11-23T03:11:33.5756021Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5756110Z Traceback (most recent call last): 2022-11-23T03:11:33.5756390Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5756474Z getattr(self, test_name)() 2022-11-23T03:11:33.5756744Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5756806Z fn() 2022-11-23T03:11:33.5757078Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5757155Z return func(*args, **kwargs) 2022-11-23T03:11:33.5757330Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5757403Z self._dist_train() 2022-11-23T03:11:33.5757568Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5757747Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5757895Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5757988Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5758124Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5758223Z raise self.failureException(msg) 2022-11-23T03:11:33.5758314Z AssertionError: False is not true 2022-11-23T03:11:33.5758405Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5758476Z dist init r=0, world=2 2022-11-23T03:11:33.5758618Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5758686Z ERROR (3.632s) 2022-11-23T03:11:33.5758985Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... test_forward_overlap errored - num_retries_left: 1 2022-11-23T03:11:33.5759072Z Traceback (most recent call last): 2022-11-23T03:11:33.5759344Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper 2022-11-23T03:11:33.5759462Z self._join_processes(fn) 2022-11-23T03:11:33.5759749Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes 2022-11-23T03:11:33.5759849Z self._check_return_codes(elapsed_time) 2022-11-23T03:11:33.5760216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes 2022-11-23T03:11:33.5760460Z raise RuntimeError(error) 2022-11-23T03:11:33.5760591Z RuntimeError: Process 1 exited with error code 10 and exception: 2022-11-23T03:11:33.5760689Z Traceback (most recent call last): 2022-11-23T03:11:33.5760985Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5761072Z getattr(self, test_name)() 2022-11-23T03:11:33.5761362Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5761427Z fn() 2022-11-23T03:11:33.5761712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5761799Z return func(*args, **kwargs) 2022-11-23T03:11:33.5761988Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5762119Z self._dist_train() 2022-11-23T03:11:33.5762295Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5762486Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5762651Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5762748Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5762889Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5762989Z raise self.failureException(msg) 2022-11-23T03:11:33.5763075Z AssertionError: False is not true 2022-11-23T03:11:33.5763091Z 2022-11-23T03:11:33.5763096Z 2022-11-23T03:11:33.5763100Z 2022-11-23T03:11:33.5763268Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52303 2022-11-23T03:11:33.5763420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52304 2022-11-23T03:11:33.5763725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5763853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5764157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5764299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5764484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:11:33.5764780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:11:33.5764910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:11:33.5765212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:11:33.5765353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:11:33.5765539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:11:33.5765859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5766178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:11:33.5766349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:11:33.5766519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:11:33.5767413Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5767494Z warnings.warn( 2022-11-23T03:11:33.5767646Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5767741Z Traceback (most recent call last): 2022-11-23T03:11:33.5768037Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5768122Z getattr(self, test_name)() 2022-11-23T03:11:33.5768415Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5768485Z fn() 2022-11-23T03:11:33.5768772Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5768850Z return func(*args, **kwargs) 2022-11-23T03:11:33.5769071Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5769149Z self._dist_train() 2022-11-23T03:11:33.5769325Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5769513Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5769679Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5769777Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5769918Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5770020Z raise self.failureException(msg) 2022-11-23T03:11:33.5770112Z AssertionError: False is not true 2022-11-23T03:11:33.5770206Z exiting process 1 with exit code: 10 2022-11-23T03:11:33.5771057Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:11:33.5771136Z warnings.warn( 2022-11-23T03:11:33.5771284Z ERROR:torch.testing._internal.common_distributed:Caught exception: 2022-11-23T03:11:33.5771378Z Traceback (most recent call last): 2022-11-23T03:11:33.5771672Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:11:33.5771760Z getattr(self, test_name)() 2022-11-23T03:11:33.5772055Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:11:33.5772224Z fn() 2022-11-23T03:11:33.5772490Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:11:33.5772574Z return func(*args, **kwargs) 2022-11-23T03:11:33.5772754Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap 2022-11-23T03:11:33.5772828Z self._dist_train() 2022-11-23T03:11:33.5772997Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train 2022-11-23T03:11:33.5773177Z e2 = run(0, sleep_cycles) # no compute, only all-gather 2022-11-23T03:11:33.5773335Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run 2022-11-23T03:11:33.5773473Z self.assertTrue(all_gather_called) 2022-11-23T03:11:33.5773605Z File "/opt/conda/lib/python3.8/unittest/case.py", line 765, in assertTrue 2022-11-23T03:11:33.5773700Z raise self.failureException(msg) 2022-11-23T03:11:33.5773790Z AssertionError: False is not true 2022-11-23T03:11:33.5773879Z exiting process 0 with exit code: 10 2022-11-23T03:11:33.5773952Z dist init r=1, world=2 2022-11-23T03:11:33.5774021Z dist init r=0, world=2 2022-11-23T03:11:33.5774165Z Process 1 terminated with exit code 10, terminating remaining processes. 2022-11-23T03:11:33.5774234Z ERROR (3.832s) 2022-11-23T03:11:33.5774396Z test_forward_overlap errored - num_retries_left: 0 2022-11-23T03:11:33.5774402Z 2022-11-23T03:11:33.5774612Z ---------------------------------------------------------------------- 2022-11-23T03:11:33.5774690Z Ran 102 tests in 1462.712s 2022-11-23T03:11:33.5774695Z 2022-11-23T03:11:33.5774794Z OK (skipped=1, expected failures=50) 2022-11-23T03:11:33.5774799Z 2022-11-23T03:11:33.5774888Z Generating XML reports... 2022-11-23T03:11:33.5775277Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221123024709.xml 2022-11-23T03:11:33.5775658Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221123024709.xml 2022-11-23T03:11:33.5775701Z 2022-11-23T03:11:33.5776138Z ##[endgroup] 2022-11-23T03:11:33.5776506Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_overlap_ni3auq3a) 2022-11-23T03:11:33.5776511Z 2022-11-23T03:11:33.5776743Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-11-23 03:11:33.414390] 2022-11-23T03:11:33.5777187Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:33.414811] 2022-11-23T03:11:36.7362190Z 2022-11-23T03:11:36.7363699Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping 2022-11-23T03:11:36.7367251Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_lsfyl6ds) 2022-11-23T03:11:36.7369840Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping 2022-11-23T03:11:36.7370627Z 2022-11-23T03:11:36.7370996Z Running tests... 2022-11-23T03:11:36.7372152Z ---------------------------------------------------------------------- 2022-11-23T03:11:36.7373219Z test_multiple_wrapping (__main__.TestMultipleWrapping) 2022-11-23T03:11:36.7375269Z This test simulates wrapping the module after training to run inference. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.475s) 2022-11-23T03:11:36.7376342Z 2022-11-23T03:11:36.7377056Z ---------------------------------------------------------------------- 2022-11-23T03:11:36.7377910Z Ran 1 test in 0.475s 2022-11-23T03:11:36.7378304Z 2022-11-23T03:11:36.7378541Z OK (skipped=1) 2022-11-23T03:11:36.7378914Z 2022-11-23T03:11:36.7379204Z Generating XML reports... 2022-11-23T03:11:36.7380970Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123031134.xml 2022-11-23T03:11:36.7381995Z 2022-11-23T03:11:36.7382742Z ##[endgroup] 2022-11-23T03:11:36.7384518Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_lsfyl6ds) 2022-11-23T03:11:36.7385552Z 2022-11-23T03:11:36.7386369Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-11-23 03:11:36.736316] 2022-11-23T03:11:36.7388566Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:36.736846] 2022-11-23T03:11:40.2956589Z 2022-11-23T03:11:40.2957885Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision 2022-11-23T03:11:40.2960596Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_44d3av2_) 2022-11-23T03:11:40.2962985Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision 2022-11-23T03:11:40.2963772Z 2022-11-23T03:11:40.2964019Z Running tests... 2022-11-23T03:11:40.2965144Z ---------------------------------------------------------------------- 2022-11-23T03:11:40.2967342Z test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.022s) 2022-11-23T03:11:40.2970376Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.2974090Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.2977185Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.2980153Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.2983191Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.2986190Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.2989213Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.2992193Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.2995244Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.2998231Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3001385Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3004415Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3007817Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3010812Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3013825Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3016806Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3020056Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3023170Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3026350Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3029495Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3032670Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3035782Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3038969Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3042197Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3045335Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3048397Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3051497Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3054704Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3057816Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3060874Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3063992Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3067157Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3069970Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3071630Z test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded) 2022-11-23T03:11:40.3072841Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s) 2022-11-23T03:11:40.3075089Z test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:11:40.3077774Z test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:11:40.3080465Z test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3083111Z test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3084693Z test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3085709Z test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3086718Z test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3087756Z test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3088823Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:40.3089328Z 2022-11-23T03:11:40.3089626Z ---------------------------------------------------------------------- 2022-11-23T03:11:40.3089958Z Ran 44 tests in 0.061s 2022-11-23T03:11:40.3090118Z 2022-11-23T03:11:40.3090217Z OK (skipped=44) 2022-11-23T03:11:40.3090369Z 2022-11-23T03:11:40.3090487Z Generating XML reports... 2022-11-23T03:11:40.3091221Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123031138.xml 2022-11-23T03:11:40.3092212Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123031138.xml 2022-11-23T03:11:40.3092640Z 2022-11-23T03:11:40.3092992Z ##[endgroup] 2022-11-23T03:11:40.3093684Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_44d3av2_) 2022-11-23T03:11:40.3094078Z 2022-11-23T03:11:40.3094374Z Running distributed/fsdp/test_fsdp_meta ... [2022-11-23 03:11:40.296358] 2022-11-23T03:11:40.3095163Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:40.297227] 2022-11-23T03:11:43.8662760Z 2022-11-23T03:11:43.8664972Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta 2022-11-23T03:11:43.8667047Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_meta_1b0d2ik_) 2022-11-23T03:11:43.8668169Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta 2022-11-23T03:11:43.8668526Z 2022-11-23T03:11:43.8668685Z Running tests... 2022-11-23T03:11:43.8669223Z ---------------------------------------------------------------------- 2022-11-23T03:11:43.8670209Z test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.514s) 2022-11-23T03:11:43.8671103Z test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T03:11:43.8672338Z test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:43.8673664Z test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:43.8674998Z test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:43.8676322Z test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:43.8677323Z test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T03:11:43.8678269Z test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T03:11:43.8679205Z test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T03:11:43.8680118Z test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T03:11:43.8681653Z test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:43.8682917Z test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:43.8683864Z test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T03:11:43.8684737Z test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T03:11:43.8685188Z 2022-11-23T03:11:43.8685547Z ---------------------------------------------------------------------- 2022-11-23T03:11:43.8685964Z Ran 14 tests in 0.523s 2022-11-23T03:11:43.8686167Z 2022-11-23T03:11:43.8686290Z OK (skipped=14) 2022-11-23T03:11:43.8686476Z 2022-11-23T03:11:43.8686620Z Generating XML reports... 2022-11-23T03:11:43.8687442Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123031141.xml 2022-11-23T03:11:43.8687911Z 2022-11-23T03:11:43.8688375Z ##[endgroup] 2022-11-23T03:11:43.8689154Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_meta_1b0d2ik_) 2022-11-23T03:11:43.8689591Z 2022-11-23T03:11:43.8689952Z Running distributed/fsdp/test_fsdp_input ... [2022-11-23 03:11:43.866858] 2022-11-23T03:11:43.8690949Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:43.867734] 2022-11-23T03:11:47.4285968Z 2022-11-23T03:11:47.4286899Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_input 2022-11-23T03:11:47.4289372Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_input_o7i7pecf) 2022-11-23T03:11:47.4291566Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input 2022-11-23T03:11:47.4292323Z 2022-11-23T03:11:47.4292571Z Running tests... 2022-11-23T03:11:47.4293686Z ---------------------------------------------------------------------- 2022-11-23T03:11:47.4294648Z test_input_type_dict (__main__.TestInput) 2022-11-23T03:11:47.4296569Z Test FSDP with input being a list or a dict, only single GPU. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.533s) 2022-11-23T03:11:47.4297905Z test_input_type_list (__main__.TestInput) 2022-11-23T03:11:47.4299749Z Test FSDP with input being a list or a dict, only single GPU. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:11:47.4300722Z 2022-11-23T03:11:47.4301462Z ---------------------------------------------------------------------- 2022-11-23T03:11:47.4302294Z Ran 2 tests in 0.536s 2022-11-23T03:11:47.4302693Z 2022-11-23T03:11:47.4302935Z OK (skipped=2) 2022-11-23T03:11:47.4303306Z 2022-11-23T03:11:47.4303606Z Generating XML reports... 2022-11-23T03:11:47.4305176Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221123031145.xml 2022-11-23T03:11:47.4306058Z 2022-11-23T03:11:47.4306806Z ##[endgroup] 2022-11-23T03:11:47.4308436Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_input_o7i7pecf) 2022-11-23T03:11:47.4309356Z 2022-11-23T03:11:47.4310121Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-11-23 03:11:47.429240] 2022-11-23T03:11:47.4312218Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:47.430111] 2022-11-23T03:11:50.8461591Z 2022-11-23T03:11:50.8462535Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_grad_acc 2022-11-23T03:11:50.8465073Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_700ouaf0) 2022-11-23T03:11:50.8467275Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc 2022-11-23T03:11:50.8468022Z 2022-11-23T03:11:50.8468274Z Running tests... 2022-11-23T03:11:50.8469407Z ---------------------------------------------------------------------- 2022-11-23T03:11:50.8471286Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc) 2022-11-23T03:11:50.8473862Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.478s) 2022-11-23T03:11:50.8476312Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc) 2022-11-23T03:11:50.8478794Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8481289Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc) 2022-11-23T03:11:50.8483665Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8485645Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc) 2022-11-23T03:11:50.8488004Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8489944Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc) 2022-11-23T03:11:50.8492256Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8494173Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc) 2022-11-23T03:11:50.8496497Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8498428Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc) 2022-11-23T03:11:50.8500767Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8502684Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc) 2022-11-23T03:11:50.8505271Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8507216Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc) 2022-11-23T03:11:50.8509519Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8511421Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc) 2022-11-23T03:11:50.8513712Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8515644Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc) 2022-11-23T03:11:50.8518075Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8519993Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc) 2022-11-23T03:11:50.8522398Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:50.8523295Z 2022-11-23T03:11:50.8524036Z ---------------------------------------------------------------------- 2022-11-23T03:11:50.8524910Z Ran 12 tests in 0.490s 2022-11-23T03:11:50.8525314Z 2022-11-23T03:11:50.8525560Z OK (skipped=12) 2022-11-23T03:11:50.8525942Z 2022-11-23T03:11:50.8526229Z Generating XML reports... 2022-11-23T03:11:50.8527839Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123031149.xml 2022-11-23T03:11:50.8528734Z 2022-11-23T03:11:50.8529498Z ##[endgroup] 2022-11-23T03:11:50.8531173Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_700ouaf0) 2022-11-23T03:11:50.8532101Z 2022-11-23T03:11:50.8532905Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-11-23 03:11:50.846752] 2022-11-23T03:11:50.8535072Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:50.847607] 2022-11-23T03:11:54.4066020Z 2022-11-23T03:11:54.4066889Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_freezing_weights 2022-11-23T03:11:54.4069371Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_szz5emm6) 2022-11-23T03:11:54.4071668Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights 2022-11-23T03:11:54.4072465Z 2022-11-23T03:11:54.4072712Z Running tests... 2022-11-23T03:11:54.4073829Z ---------------------------------------------------------------------- 2022-11-23T03:11:54.4076733Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.488s) 2022-11-23T03:11:54.4080025Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:54.4084284Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:54.4087689Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:54.4090911Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:54.4094313Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:54.4097552Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:54.4100767Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.010s) 2022-11-23T03:11:54.4102067Z 2022-11-23T03:11:54.4102815Z ---------------------------------------------------------------------- 2022-11-23T03:11:54.4103677Z Ran 8 tests in 0.504s 2022-11-23T03:11:54.4104084Z 2022-11-23T03:11:54.4104325Z OK (skipped=8) 2022-11-23T03:11:54.4104697Z 2022-11-23T03:11:54.4104965Z Generating XML reports... 2022-11-23T03:11:54.4106719Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221123031152.xml 2022-11-23T03:11:54.4107720Z 2022-11-23T03:11:54.4108498Z ##[endgroup] 2022-11-23T03:11:54.4110277Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_szz5emm6) 2022-11-23T03:11:54.4111292Z 2022-11-23T03:11:54.4112086Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-11-23 03:11:54.407215] 2022-11-23T03:11:54.4114198Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:54.408129] 2022-11-23T03:11:57.9641337Z 2022-11-23T03:11:57.9642480Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_exec_order 2022-11-23T03:11:57.9644890Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_exec_order_8guyu2fc) 2022-11-23T03:11:57.9647134Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order 2022-11-23T03:11:57.9647892Z 2022-11-23T03:11:57.9648144Z Running tests... 2022-11-23T03:11:57.9649274Z ---------------------------------------------------------------------- 2022-11-23T03:11:57.9650674Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2022-11-23T03:11:57.9653086Z Tests that FSDP errors if the all-gather order differs across ranks ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.495s) 2022-11-23T03:11:57.9655249Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2022-11-23T03:11:57.9657489Z Tests that FSDP errors if the all-gather order differs across ranks ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:57.9659265Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-11-23T03:11:57.9661827Z Tests that FSDP warns the user if the all-gather order changes after ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:57.9663570Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-11-23T03:11:57.9665816Z Tests that FSDP warns the user if the all-gather order changes after ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:57.9667557Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-11-23T03:11:57.9670035Z Tests that FSDP warns the user if the all-gather order changes after ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:57.9671790Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-11-23T03:11:57.9674047Z Tests that FSDP warns the user if the all-gather order changes after ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:57.9676645Z test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:57.9679410Z test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:11:57.9680674Z 2022-11-23T03:11:57.9681416Z ---------------------------------------------------------------------- 2022-11-23T03:11:57.9682311Z Ran 8 tests in 0.504s 2022-11-23T03:11:57.9682716Z 2022-11-23T03:11:57.9682957Z OK (skipped=8) 2022-11-23T03:11:57.9683332Z 2022-11-23T03:11:57.9683621Z Generating XML reports... 2022-11-23T03:11:57.9685296Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123031156.xml 2022-11-23T03:11:57.9686240Z 2022-11-23T03:11:57.9687249Z ##[endgroup] 2022-11-23T03:11:57.9688915Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_exec_order_8guyu2fc) 2022-11-23T03:11:57.9689880Z 2022-11-23T03:11:57.9690651Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2022-11-23 03:11:57.964727] 2022-11-23T03:11:57.9692781Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:57.965601] 2022-11-23T03:12:01.5579888Z 2022-11-23T03:12:01.5580872Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm_hooks 2022-11-23T03:12:01.5583348Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_jiwn2mzv) 2022-11-23T03:12:01.5585620Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks 2022-11-23T03:12:01.5586382Z 2022-11-23T03:12:01.5587139Z Running tests... 2022-11-23T03:12:01.5588281Z ---------------------------------------------------------------------- 2022-11-23T03:12:01.5589804Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:12:01.5591665Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:12:01.5593493Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:12:01.5595320Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:12:01.5597119Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:12:01.5598921Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:12:01.5601470Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5604654Z Tests FSDP's default communication hook's behavior and correctness. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.511s) 2022-11-23T03:12:01.5606415Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5608693Z Tests FSDP's default communication hook's behavior and correctness. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T03:12:01.5633797Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5636492Z Tests FSDP's default communication hook's behavior and correctness. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5638412Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5641348Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5643321Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5645568Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5647408Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5649675Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5651487Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5653743Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5655521Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5658077Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5659874Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5662128Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5664820Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:01.5667757Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:01.5670877Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:01.5673842Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:01.5676741Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:01.5679648Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:01.5681652Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5683803Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:01.5685495Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5687664Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:01.5689366Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5691548Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:01.5693217Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5695376Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5697032Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5699175Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5700864Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:12:01.5703149Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:01.5704142Z 2022-11-23T03:12:01.5704876Z ---------------------------------------------------------------------- 2022-11-23T03:12:01.5705736Z Ran 27 tests in 0.547s 2022-11-23T03:12:01.5706144Z 2022-11-23T03:12:01.5706392Z OK (skipped=27) 2022-11-23T03:12:01.5706774Z 2022-11-23T03:12:01.5707070Z Generating XML reports... 2022-11-23T03:12:01.5708825Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221123031159.xml 2022-11-23T03:12:01.5709831Z 2022-11-23T03:12:01.5710686Z ##[endgroup] 2022-11-23T03:12:01.5712361Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_jiwn2mzv) 2022-11-23T03:12:01.5713310Z 2022-11-23T03:12:01.5714150Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-11-23 03:12:01.558672] 2022-11-23T03:12:01.5716298Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:01.559527] 2022-11-23T03:12:05.1130200Z 2022-11-23T03:12:05.1131702Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_clip_grad_norm 2022-11-23T03:12:05.1134350Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_s1xwy4a2) 2022-11-23T03:12:05.1136619Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm 2022-11-23T03:12:05.1137597Z 2022-11-23T03:12:05.1137855Z Running tests... 2022-11-23T03:12:05.1139010Z ---------------------------------------------------------------------- 2022-11-23T03:12:05.1140017Z test_ddp_parity (__main__.TestClipGradNorm) 2022-11-23T03:12:05.1142103Z Tests FSDP with ``FullyShardedDataParallel.clip_grad_norm_()` against ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.531s) 2022-11-23T03:12:05.1143553Z test_non_root (__main__.TestClipGradNorm) 2022-11-23T03:12:05.1145542Z Tests that calling ``clip_grad_norm_()`` on a non-root FSDP instance ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:05.1146623Z 2022-11-23T03:12:05.1147371Z ---------------------------------------------------------------------- 2022-11-23T03:12:05.1148228Z Ran 2 tests in 0.533s 2022-11-23T03:12:05.1148630Z 2022-11-23T03:12:05.1148856Z OK (skipped=2) 2022-11-23T03:12:05.1149225Z 2022-11-23T03:12:05.1149519Z Generating XML reports... 2022-11-23T03:12:05.1151242Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221123031203.xml 2022-11-23T03:12:05.1152206Z 2022-11-23T03:12:05.1152956Z ##[endgroup] 2022-11-23T03:12:05.1154707Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_s1xwy4a2) 2022-11-23T03:12:05.1155686Z 2022-11-23T03:12:05.1156453Z Running distributed/fsdp/test_fsdp_apply ... [2022-11-23 03:12:05.113630] 2022-11-23T03:12:05.1158573Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:05.114477] 2022-11-23T03:12:08.6660960Z 2022-11-23T03:12:08.6661878Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply 2022-11-23T03:12:08.6664197Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_apply_7x1z1e9b) 2022-11-23T03:12:08.6666376Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply 2022-11-23T03:12:08.6667700Z 2022-11-23T03:12:08.6668115Z Running tests... 2022-11-23T03:12:08.6669286Z ---------------------------------------------------------------------- 2022-11-23T03:12:08.6670318Z test_apply_in_summon_raises_error (__main__.TestApply) 2022-11-23T03:12:08.6672335Z Tests that calling ``apply()`` on an FSDP instance inside the ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.493s) 2022-11-23T03:12:08.6673710Z test_nested_module_apply (__main__.TestApply) 2022-11-23T03:12:08.6675639Z Tests that ``apply()`` modifies parameter values in-place on a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:08.6677209Z test_transformer_module_apply (__main__.TestApply) 2022-11-23T03:12:08.6679147Z Tests that ``apply()`` modifies parameter values in-place on an ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:12:08.6680381Z 2022-11-23T03:12:08.6681129Z ---------------------------------------------------------------------- 2022-11-23T03:12:08.6681993Z Ran 3 tests in 0.495s 2022-11-23T03:12:08.6682392Z 2022-11-23T03:12:08.6682636Z OK (skipped=3) 2022-11-23T03:12:08.6683268Z 2022-11-23T03:12:08.6683569Z Generating XML reports... 2022-11-23T03:12:08.6685161Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123031206.xml 2022-11-23T03:12:08.6686038Z 2022-11-23T03:12:08.6686816Z ##[endgroup] 2022-11-23T03:12:08.6688462Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_apply_7x1z1e9b) 2022-11-23T03:12:08.6689403Z 2022-11-23T03:12:08.6690243Z Running distributed/fsdp/test_checkpoint_wrapper ... [2022-11-23 03:12:08.666684] 2022-11-23T03:12:08.6692441Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:08.667557] 2022-11-23T03:12:14.6563850Z 2022-11-23T03:12:14.6564816Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper 2022-11-23T03:12:14.6567262Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_3ct40kij) 2022-11-23T03:12:14.6569568Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper 2022-11-23T03:12:14.6570372Z 2022-11-23T03:12:14.6570798Z Running tests... 2022-11-23T03:12:14.6571981Z ---------------------------------------------------------------------- 2022-11-23T03:12:14.6573104Z test_apply_activation_checkpointing (__main__.CheckpointWrapperTest) 2022-11-23T03:12:14.6575163Z Ensures that `apply_activation_checkpointing` can be used ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.492s) 2022-11-23T03:12:14.6577674Z test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:14.6580398Z test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:14.6581955Z test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest) 2022-11-23T03:12:14.6583036Z Tests that using checkpoint_wrapper or the functional ... ok (2.020s) 2022-11-23T03:12:14.6585123Z test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T03:12:14.6587483Z test_fqn (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:14.6590383Z test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:14.6591455Z 2022-11-23T03:12:14.6592200Z ---------------------------------------------------------------------- 2022-11-23T03:12:14.6593072Z Ran 7 tests in 2.522s 2022-11-23T03:12:14.6593456Z 2022-11-23T03:12:14.6593709Z OK (skipped=6) 2022-11-23T03:12:14.6594088Z 2022-11-23T03:12:14.6594386Z Generating XML reports... 2022-11-23T03:12:14.6596160Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123031210.xml 2022-11-23T03:12:14.6597178Z 2022-11-23T03:12:14.6597947Z ##[endgroup] 2022-11-23T03:12:14.6599722Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_3ct40kij) 2022-11-23T03:12:14.6601149Z 2022-11-23T03:12:14.6601932Z Running distributed/elastic/utils/logging_test ... [2022-11-23 03:12:14.656956] 2022-11-23T03:12:14.6604289Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:14.657818] 2022-11-23T03:12:18.2150942Z 2022-11-23T03:12:18.2152233Z Expand the folded group to see the log file of distributed/elastic/utils/logging_test 2022-11-23T03:12:18.2154575Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-logging_test__8s4qcit) 2022-11-23T03:12:18.2156848Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test 2022-11-23T03:12:18.2157892Z 2022-11-23T03:12:18.2158150Z Running tests... 2022-11-23T03:12:18.2159298Z ---------------------------------------------------------------------- 2022-11-23T03:12:18.2161943Z test_derive_module_name (__main__.LoggingTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.529s) 2022-11-23T03:12:18.2164213Z test_logger_name (__main__.LoggingTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:18.2165139Z 2022-11-23T03:12:18.2165871Z ---------------------------------------------------------------------- 2022-11-23T03:12:18.2166887Z Ran 2 tests in 0.532s 2022-11-23T03:12:18.2167294Z 2022-11-23T03:12:18.2167544Z OK (skipped=2) 2022-11-23T03:12:18.2167918Z 2022-11-23T03:12:18.2168212Z Generating XML reports... 2022-11-23T03:12:18.2169883Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221123031216.xml 2022-11-23T03:12:18.2170822Z 2022-11-23T03:12:18.2171589Z ##[endgroup] 2022-11-23T03:12:18.2173332Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-logging_test__8s4qcit) 2022-11-23T03:12:18.2174337Z 2022-11-23T03:12:18.2175160Z Running distributed/elastic/timer/local_timer_test ... [2022-11-23 03:12:18.215637] 2022-11-23T03:12:18.2177354Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:18.216545] 2022-11-23T03:12:21.8722318Z 2022-11-23T03:12:21.8723298Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_test 2022-11-23T03:12:21.8726248Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-local_timer_test_p6npx8gc) 2022-11-23T03:12:21.8729845Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test 2022-11-23T03:12:21.8732185Z 2022-11-23T03:12:21.8732478Z Running tests... 2022-11-23T03:12:21.8733648Z ---------------------------------------------------------------------- 2022-11-23T03:12:21.8734682Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8735471Z tests that: ... ok (0.533s) 2022-11-23T03:12:21.8736308Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8737589Z tests that: ... test_acquire_release succeeded - num_retries_left: 50 2022-11-23T03:12:21.8738389Z ok (0.003s) 2022-11-23T03:12:21.8739140Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8740378Z tests that: ... test_acquire_release succeeded - num_retries_left: 49 2022-11-23T03:12:21.8741131Z ok (0.002s) 2022-11-23T03:12:21.8741880Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8743123Z tests that: ... test_acquire_release succeeded - num_retries_left: 48 2022-11-23T03:12:21.8743917Z ok (0.002s) 2022-11-23T03:12:21.8744674Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8745896Z tests that: ... test_acquire_release succeeded - num_retries_left: 47 2022-11-23T03:12:21.8746649Z ok (0.003s) 2022-11-23T03:12:21.8747399Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8748861Z tests that: ... test_acquire_release succeeded - num_retries_left: 46 2022-11-23T03:12:21.8749645Z ok (0.002s) 2022-11-23T03:12:21.8750399Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8751631Z tests that: ... test_acquire_release succeeded - num_retries_left: 45 2022-11-23T03:12:21.8752374Z ok (0.002s) 2022-11-23T03:12:21.8753299Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8754530Z tests that: ... test_acquire_release succeeded - num_retries_left: 44 2022-11-23T03:12:21.8755278Z ok (0.002s) 2022-11-23T03:12:21.8756036Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8757272Z tests that: ... test_acquire_release succeeded - num_retries_left: 43 2022-11-23T03:12:21.8758044Z ok (0.002s) 2022-11-23T03:12:21.8758785Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8759998Z tests that: ... test_acquire_release succeeded - num_retries_left: 42 2022-11-23T03:12:21.8760916Z ok (0.002s) 2022-11-23T03:12:21.8761673Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8762908Z tests that: ... test_acquire_release succeeded - num_retries_left: 41 2022-11-23T03:12:21.8763680Z ok (0.002s) 2022-11-23T03:12:21.8764434Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8765663Z tests that: ... test_acquire_release succeeded - num_retries_left: 40 2022-11-23T03:12:21.8766401Z ok (0.002s) 2022-11-23T03:12:21.8767147Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8768362Z tests that: ... test_acquire_release succeeded - num_retries_left: 39 2022-11-23T03:12:21.8769152Z ok (0.004s) 2022-11-23T03:12:21.8769898Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8771117Z tests that: ... test_acquire_release succeeded - num_retries_left: 38 2022-11-23T03:12:21.8771868Z ok (0.002s) 2022-11-23T03:12:21.8772619Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8773858Z tests that: ... test_acquire_release succeeded - num_retries_left: 37 2022-11-23T03:12:21.8774639Z ok (0.002s) 2022-11-23T03:12:21.8775390Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8776578Z tests that: ... test_acquire_release succeeded - num_retries_left: 36 2022-11-23T03:12:21.8777338Z ok (0.002s) 2022-11-23T03:12:21.8778086Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8779295Z tests that: ... test_acquire_release succeeded - num_retries_left: 35 2022-11-23T03:12:21.8780072Z ok (0.002s) 2022-11-23T03:12:21.8780814Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8782193Z tests that: ... test_acquire_release succeeded - num_retries_left: 34 2022-11-23T03:12:21.8782956Z ok (0.002s) 2022-11-23T03:12:21.8783707Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8784936Z tests that: ... test_acquire_release succeeded - num_retries_left: 33 2022-11-23T03:12:21.8785702Z ok (0.002s) 2022-11-23T03:12:21.8786460Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8787664Z tests that: ... test_acquire_release succeeded - num_retries_left: 32 2022-11-23T03:12:21.8788422Z ok (0.002s) 2022-11-23T03:12:21.8789169Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8790381Z tests that: ... test_acquire_release succeeded - num_retries_left: 31 2022-11-23T03:12:21.8791156Z ok (0.002s) 2022-11-23T03:12:21.8791901Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8793097Z tests that: ... test_acquire_release succeeded - num_retries_left: 30 2022-11-23T03:12:21.8793868Z ok (0.002s) 2022-11-23T03:12:21.8794613Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8795832Z tests that: ... test_acquire_release succeeded - num_retries_left: 29 2022-11-23T03:12:21.8796597Z ok (0.002s) 2022-11-23T03:12:21.8797349Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8798670Z tests that: ... test_acquire_release succeeded - num_retries_left: 28 2022-11-23T03:12:21.8799445Z ok (0.002s) 2022-11-23T03:12:21.8800345Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8801578Z tests that: ... test_acquire_release succeeded - num_retries_left: 27 2022-11-23T03:12:21.8802338Z ok (0.002s) 2022-11-23T03:12:21.8803085Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8804279Z tests that: ... test_acquire_release succeeded - num_retries_left: 26 2022-11-23T03:12:21.8805046Z ok (0.002s) 2022-11-23T03:12:21.8805796Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8807044Z tests that: ... test_acquire_release succeeded - num_retries_left: 25 2022-11-23T03:12:21.8807807Z ok (0.002s) 2022-11-23T03:12:21.8808553Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8809757Z tests that: ... test_acquire_release succeeded - num_retries_left: 24 2022-11-23T03:12:21.8810528Z ok (0.002s) 2022-11-23T03:12:21.8811280Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8812504Z tests that: ... test_acquire_release succeeded - num_retries_left: 23 2022-11-23T03:12:21.8813271Z ok (0.002s) 2022-11-23T03:12:21.8814009Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8815199Z tests that: ... test_acquire_release succeeded - num_retries_left: 22 2022-11-23T03:12:21.8815969Z ok (0.002s) 2022-11-23T03:12:21.8816724Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8817954Z tests that: ... test_acquire_release succeeded - num_retries_left: 21 2022-11-23T03:12:21.8818722Z ok (0.002s) 2022-11-23T03:12:21.8819448Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8820661Z tests that: ... test_acquire_release succeeded - num_retries_left: 20 2022-11-23T03:12:21.8821430Z ok (0.002s) 2022-11-23T03:12:21.8822186Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8823418Z tests that: ... test_acquire_release succeeded - num_retries_left: 19 2022-11-23T03:12:21.8824178Z ok (0.002s) 2022-11-23T03:12:21.8824901Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8826106Z tests that: ... test_acquire_release succeeded - num_retries_left: 18 2022-11-23T03:12:21.8826864Z ok (0.002s) 2022-11-23T03:12:21.8827610Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8828833Z tests that: ... test_acquire_release succeeded - num_retries_left: 17 2022-11-23T03:12:21.8829590Z ok (0.003s) 2022-11-23T03:12:21.8830315Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8831695Z tests that: ... test_acquire_release succeeded - num_retries_left: 16 2022-11-23T03:12:21.8832466Z ok (0.002s) 2022-11-23T03:12:21.8833215Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8834436Z tests that: ... test_acquire_release succeeded - num_retries_left: 15 2022-11-23T03:12:21.8835213Z ok (0.002s) 2022-11-23T03:12:21.8835941Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8837152Z tests that: ... test_acquire_release succeeded - num_retries_left: 14 2022-11-23T03:12:21.8837926Z ok (0.002s) 2022-11-23T03:12:21.8838678Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8839903Z tests that: ... test_acquire_release succeeded - num_retries_left: 13 2022-11-23T03:12:21.8840876Z ok (0.003s) 2022-11-23T03:12:21.8841164Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8841693Z tests that: ... test_acquire_release succeeded - num_retries_left: 12 2022-11-23T03:12:21.8842006Z ok (0.002s) 2022-11-23T03:12:21.8842310Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8842793Z tests that: ... test_acquire_release succeeded - num_retries_left: 11 2022-11-23T03:12:21.8843105Z ok (0.002s) 2022-11-23T03:12:21.8843392Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8843938Z tests that: ... test_acquire_release succeeded - num_retries_left: 10 2022-11-23T03:12:21.8844249Z ok (0.002s) 2022-11-23T03:12:21.8844551Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8845034Z tests that: ... test_acquire_release succeeded - num_retries_left: 9 2022-11-23T03:12:21.8845345Z ok (0.002s) 2022-11-23T03:12:21.8845634Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8846118Z tests that: ... test_acquire_release succeeded - num_retries_left: 8 2022-11-23T03:12:21.8846428Z ok (0.002s) 2022-11-23T03:12:21.8846734Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8847226Z tests that: ... test_acquire_release succeeded - num_retries_left: 7 2022-11-23T03:12:21.8847540Z ok (0.002s) 2022-11-23T03:12:21.8847825Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8848307Z tests that: ... test_acquire_release succeeded - num_retries_left: 6 2022-11-23T03:12:21.8848619Z ok (0.002s) 2022-11-23T03:12:21.8848920Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8849402Z tests that: ... test_acquire_release succeeded - num_retries_left: 5 2022-11-23T03:12:21.8849714Z ok (0.002s) 2022-11-23T03:12:21.8850002Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8850483Z tests that: ... test_acquire_release succeeded - num_retries_left: 4 2022-11-23T03:12:21.8850790Z ok (0.003s) 2022-11-23T03:12:21.8851090Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8851571Z tests that: ... test_acquire_release succeeded - num_retries_left: 3 2022-11-23T03:12:21.8851867Z ok (0.002s) 2022-11-23T03:12:21.8852168Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8852651Z tests that: ... test_acquire_release succeeded - num_retries_left: 2 2022-11-23T03:12:21.8852958Z ok (0.002s) 2022-11-23T03:12:21.8853257Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8853738Z tests that: ... test_acquire_release succeeded - num_retries_left: 1 2022-11-23T03:12:21.8854033Z ok (0.002s) 2022-11-23T03:12:21.8854342Z test_expired_timers (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8854969Z tests that a single expired timer on a process should terminate ... test_acquire_release succeeded - num_retries_left: 0 2022-11-23T03:12:21.8855675Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:21.8856141Z test_valid_timers (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8856946Z tests that valid timers are processed correctly and the process is left alone ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:21.8857589Z test_watchdog_call_count (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8858363Z checks that the watchdog function ran wait/interval +- 1 times ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:21.8858920Z test_watchdog_empty_queue (__main__.LocalTimerServerTest) 2022-11-23T03:12:21.8859655Z checks that the watchdog can run on an empty queue ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:12:21.8860538Z test_client_interaction (__main__.LocalTimerTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:21.8861437Z test_exception_propagation (__main__.LocalTimerTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:12:21.8861974Z test_get_timer_recursive (__main__.LocalTimerTest) 2022-11-23T03:12:21.8862770Z If a function acquires a countdown timer with default scope, ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:21.8863662Z test_happy_path (__main__.LocalTimerTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:12:21.8864515Z test_no_client (__main__.LocalTimerTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.000s) 2022-11-23T03:12:21.8865350Z test_timer (__main__.LocalTimerTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:21.8866253Z test_get (__main__.MultiprocessingRequestQueueTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:21.8866843Z test_get_less_than_size (__main__.MultiprocessingRequestQueueTest) 2022-11-23T03:12:21.8867554Z Tests slow producer. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:21.8868081Z test_get_size (__main__.MultiprocessingRequestQueueTest) 2022-11-23T03:12:21.8868857Z Creates a "producer" process that enqueues ``n`` elements ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:21.8869235Z 2022-11-23T03:12:21.8869526Z ---------------------------------------------------------------------- 2022-11-23T03:12:21.8869870Z Ran 64 tests in 0.671s 2022-11-23T03:12:21.8870034Z 2022-11-23T03:12:21.8870135Z OK (skipped=13) 2022-11-23T03:12:21.8870290Z 2022-11-23T03:12:21.8870412Z Generating XML reports... 2022-11-23T03:12:21.8871091Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20221123031219.xml 2022-11-23T03:12:21.8872037Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20221123031219.xml 2022-11-23T03:12:21.8872996Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20221123031219.xml 2022-11-23T03:12:21.8873439Z 2022-11-23T03:12:21.8873815Z ##[endgroup] 2022-11-23T03:12:21.8874518Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-local_timer_test_p6npx8gc) 2022-11-23T03:12:21.8874920Z 2022-11-23T03:12:21.8875214Z Running distributed/elastic/timer/api_test ... [2022-11-23 03:12:21.873229] 2022-11-23T03:12:21.8876038Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:21.874082] 2022-11-23T03:12:24.5630444Z 2022-11-23T03:12:24.5631374Z Expand the folded group to see the log file of distributed/elastic/timer/api_test 2022-11-23T03:12:24.5633701Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-api_test_0at8ab1p) 2022-11-23T03:12:24.5634708Z 2022-11-23T03:12:24.5635427Z ##[endgroup] 2022-11-23T03:12:24.5637445Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-api_test_0at8ab1p) 2022-11-23T03:12:24.5638419Z 2022-11-23T03:12:24.5642793Z Running distributed/elastic/metrics/api_test ... [2022-11-23 03:12:24.563665] 2022-11-23T03:12:24.5650375Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:24.564544] 2022-11-23T03:12:28.0616725Z 2022-11-23T03:12:28.0617650Z Expand the folded group to see the log file of distributed/elastic/metrics/api_test 2022-11-23T03:12:28.0620475Z ##[group]PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-metrics-api_test_n6mvvfrd) 2022-11-23T03:12:28.0622842Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test 2022-11-23T03:12:28.0623636Z 2022-11-23T03:12:28.0623888Z Running tests... 2022-11-23T03:12:28.0625223Z ---------------------------------------------------------------------- 2022-11-23T03:12:28.0627329Z test_get_metric_name (__main__.MetricsApiTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.499s) 2022-11-23T03:12:28.0629600Z test_inheritance (__main__.MetricsApiTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:28.0631799Z test_profile (__main__.MetricsApiTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:28.0632735Z 2022-11-23T03:12:28.0633595Z ---------------------------------------------------------------------- 2022-11-23T03:12:28.0634521Z Ran 3 tests in 0.501s 2022-11-23T03:12:28.0634923Z 2022-11-23T03:12:28.0635165Z OK (skipped=3) 2022-11-23T03:12:28.0635545Z 2022-11-23T03:12:28.0635833Z Generating XML reports... 2022-11-23T03:12:28.0637512Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221123031226.xml 2022-11-23T03:12:28.0638478Z 2022-11-23T03:12:28.0639252Z ##[endgroup] 2022-11-23T03:12:28.0641652Z FINISHED PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-metrics-api_test_n6mvvfrd) 2022-11-23T03:12:28.0642650Z 2022-11-23T03:12:28.0643430Z Running distributed/checkpoint/test_utils ... [2022-11-23 03:12:28.062245] 2022-11-23T03:12:28.0645524Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:28.063127] 2022-11-23T03:12:31.5979478Z 2022-11-23T03:12:31.5981355Z Expand the folded group to see the log file of distributed/checkpoint/test_utils 2022-11-23T03:12:31.5984604Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_utils_h85tqkpc) 2022-11-23T03:12:31.5986830Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_utils 2022-11-23T03:12:31.5987596Z 2022-11-23T03:12:31.5987897Z Running tests... 2022-11-23T03:12:31.5989065Z ---------------------------------------------------------------------- 2022-11-23T03:12:31.5992168Z test_flat_data (__main__.TestMedatadaIndex) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.536s) 2022-11-23T03:12:31.5995096Z test_index_hint_ignored_on_equals (__main__.TestMedatadaIndex) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:31.5997531Z test_index_hint_ignored_on_hash (__main__.TestMedatadaIndex) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:31.6001453Z test_init_convert_offset (__main__.TestMedatadaIndex) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:12:31.6003808Z test_sharded_tensor_lookup (__main__.TestMedatadaIndex) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:31.6004792Z 2022-11-23T03:12:31.6005540Z ---------------------------------------------------------------------- 2022-11-23T03:12:31.6006409Z Ran 5 tests in 0.541s 2022-11-23T03:12:31.6006814Z 2022-11-23T03:12:31.6007056Z OK (skipped=5) 2022-11-23T03:12:31.6007440Z 2022-11-23T03:12:31.6007709Z Generating XML reports... 2022-11-23T03:12:31.6009660Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_utils/TEST-TestMedatadaIndex-20221123031229.xml 2022-11-23T03:12:31.6010627Z 2022-11-23T03:12:31.6011413Z ##[endgroup] 2022-11-23T03:12:31.6013096Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_utils_h85tqkpc) 2022-11-23T03:12:31.6014051Z 2022-11-23T03:12:31.6014843Z Running distributed/checkpoint/test_planner ... [2022-11-23 03:12:31.598575] 2022-11-23T03:12:31.6016965Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:31.599432] 2022-11-23T03:12:35.1622648Z 2022-11-23T03:12:35.1624151Z Expand the folded group to see the log file of distributed/checkpoint/test_planner 2022-11-23T03:12:35.1627652Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_planner_50evrm2l) 2022-11-23T03:12:35.1629962Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_planner 2022-11-23T03:12:35.1630735Z 2022-11-23T03:12:35.1630984Z Running tests... 2022-11-23T03:12:35.1632088Z ---------------------------------------------------------------------- 2022-11-23T03:12:35.1634011Z test_global_plan (__main__.TestSavePlan) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.501s) 2022-11-23T03:12:35.1636277Z test_load_with_resharding (__main__.TestSavePlan) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.005s) 2022-11-23T03:12:35.1638640Z test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:35.1641132Z test_local_load_plan (__main__.TestSavePlan) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:35.1643347Z test_local_plan (__main__.TestSavePlan) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:12:35.1644251Z 2022-11-23T03:12:35.1644982Z ---------------------------------------------------------------------- 2022-11-23T03:12:35.1645838Z Ran 5 tests in 0.514s 2022-11-23T03:12:35.1646245Z 2022-11-23T03:12:35.1646487Z OK (skipped=5) 2022-11-23T03:12:35.1646842Z 2022-11-23T03:12:35.1647132Z Generating XML reports... 2022-11-23T03:12:35.1648769Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20221123031233.xml 2022-11-23T03:12:35.1650144Z 2022-11-23T03:12:35.1650927Z ##[endgroup] 2022-11-23T03:12:35.1652655Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_planner_50evrm2l) 2022-11-23T03:12:35.1653657Z 2022-11-23T03:12:35.1654548Z Running distributed/checkpoint/test_file_system_checkpoint ... [2022-11-23 03:12:35.162793] 2022-11-23T03:12:35.1656823Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:35.163634] 2022-11-23T03:20:05.3130502Z 2022-11-23T03:20:05.3132501Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint 2022-11-23T03:20:05.3136674Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_2ooude1n) 2022-11-23T03:20:05.3139300Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint 2022-11-23T03:20:05.3140531Z 2022-11-23T03:20:05.3141018Z Running tests... 2022-11-23T03:20:05.3143286Z ---------------------------------------------------------------------- 2022-11-23T03:20:05.3144846Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53534 2022-11-23T03:20:05.3146366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53535 2022-11-23T03:20:05.3148736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3150612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3153163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3155181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3157474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3159515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3162482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3164421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3166917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3168870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3170770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3172762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3175655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3178701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3180252Z ok (4.355s) 2022-11-23T03:20:05.3182643Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 50 2022-11-23T03:20:05.3184653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53665 2022-11-23T03:20:05.3186417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53666 2022-11-23T03:20:05.3188929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3191003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3193415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3195321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3197115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3199107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3201916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3203720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3206492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3208124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3209304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3210584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3212593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3214519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3215548Z ok (3.832s) 2022-11-23T03:20:05.3217085Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 49 2022-11-23T03:20:05.3218486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53796 2022-11-23T03:20:05.3219668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53797 2022-11-23T03:20:05.3221362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3222532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3224121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3225369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3226528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3227843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3229573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3230755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3232318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3233585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3234747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3236051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3237876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3239774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3240991Z ok (3.934s) 2022-11-23T03:20:05.3242534Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 48 2022-11-23T03:20:05.3243908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53927 2022-11-23T03:20:05.3245267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53928 2022-11-23T03:20:05.3245925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3246401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3247025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3247520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3247983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3248487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3249172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3249650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3250273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3250768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3251278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3251794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3252504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3253251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3253659Z ok (3.935s) 2022-11-23T03:20:05.3254267Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 47 2022-11-23T03:20:05.3254824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54058 2022-11-23T03:20:05.3255294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54059 2022-11-23T03:20:05.3255950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3256423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3257033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3257527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3257988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3258506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3259199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3259669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3260296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3260772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3261230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3261749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3262459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3263206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3263666Z ok (3.733s) 2022-11-23T03:20:05.3264271Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 46 2022-11-23T03:20:05.3264814Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54189 2022-11-23T03:20:05.3265286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54190 2022-11-23T03:20:05.3265934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3266407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3267026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3267519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3267983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3268499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3269169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3269696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3270324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3270818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3271277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3271787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3272498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3273232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3273640Z ok (4.035s) 2022-11-23T03:20:05.3274244Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 45 2022-11-23T03:20:05.3274800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54320 2022-11-23T03:20:05.3275267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54321 2022-11-23T03:20:05.3275922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3276394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3277000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3277501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3277963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3278477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3279161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3279629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3280328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3280824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3281272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3281785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3282573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3283326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3283730Z ok (3.935s) 2022-11-23T03:20:05.3284332Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 44 2022-11-23T03:20:05.3284915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54451 2022-11-23T03:20:05.3285373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54452 2022-11-23T03:20:05.3286033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3286499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3287127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3287621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3288131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3288645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3289332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3289783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3290409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3290899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3291357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3291872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3292588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3293336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3293729Z ok (3.833s) 2022-11-23T03:20:05.3294332Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 43 2022-11-23T03:20:05.3294878Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54582 2022-11-23T03:20:05.3295344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54583 2022-11-23T03:20:05.3295999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3296472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3297096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3297581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3298043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3298553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3299231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3299700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3300326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3300861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3301320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3301819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3302534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3303279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3303686Z ok (3.931s) 2022-11-23T03:20:05.3304287Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 42 2022-11-23T03:20:05.3304838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54713 2022-11-23T03:20:05.3305314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54714 2022-11-23T03:20:05.3305956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3306424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3307086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3307585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3308045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3308557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3309240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3309695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3310315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3310807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3311270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3311788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3312500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3313248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3313656Z ok (4.334s) 2022-11-23T03:20:05.3314239Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 41 2022-11-23T03:20:05.3314798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54844 2022-11-23T03:20:05.3315268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54845 2022-11-23T03:20:05.3315931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3316400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3317022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3317508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3317950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3318459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3319191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3319662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3320374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3320875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3321339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3321837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3322563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3323311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3323723Z ok (3.933s) 2022-11-23T03:20:05.3324324Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 40 2022-11-23T03:20:05.3324875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54975 2022-11-23T03:20:05.3325413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54976 2022-11-23T03:20:05.3326075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3326530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3327153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3327644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3328104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3328631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3329318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3329781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3330393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3330881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3331336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3331849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3332565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3333315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3333722Z ok (4.135s) 2022-11-23T03:20:05.3334309Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 39 2022-11-23T03:20:05.3334863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55106 2022-11-23T03:20:05.3335334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55107 2022-11-23T03:20:05.3335987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3336456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3337079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3337571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3338089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3338743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3339211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3339831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3340317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3340775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3341288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3341811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3342513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3343255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3343662Z ok (3.834s) 2022-11-23T03:20:05.3344336Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 38 2022-11-23T03:20:05.3344892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55237 2022-11-23T03:20:05.3345363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55238 2022-11-23T03:20:05.3346021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3346484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3347088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3347588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3348049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3348570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3349254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3349722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3350341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3350821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3351282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3351798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3352511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3353268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3353674Z ok (3.834s) 2022-11-23T03:20:05.3354274Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 37 2022-11-23T03:20:05.3354811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55368 2022-11-23T03:20:05.3355278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55369 2022-11-23T03:20:05.3355925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3356444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3357066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3357559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3358021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3358538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3359204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3359670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3360371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3360877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3361339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3361857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3362626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3363363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3363769Z ok (3.934s) 2022-11-23T03:20:05.3364365Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 36 2022-11-23T03:20:05.3364918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55499 2022-11-23T03:20:05.3365566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55500 2022-11-23T03:20:05.3366225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3366693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3367305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3367798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3368258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3368775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3369452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3369915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3370540Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3371025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3371470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3371984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3372698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3373447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3373852Z ok (3.740s) 2022-11-23T03:20:05.3374451Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 35 2022-11-23T03:20:05.3375059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55630 2022-11-23T03:20:05.3375516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55631 2022-11-23T03:20:05.3376173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3376645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3377265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3377753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3378222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3378739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3379404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3379872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3380490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3381026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3381486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3382847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3383564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3384318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3384710Z ok (4.436s) 2022-11-23T03:20:05.3385313Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 34 2022-11-23T03:20:05.3385869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55761 2022-11-23T03:20:05.3386335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55762 2022-11-23T03:20:05.3386990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3387457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3388075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3388552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3389008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3389522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3390206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3390675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3391303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3391797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3392244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3392757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3393469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3394222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3394685Z ok (3.935s) 2022-11-23T03:20:05.3395292Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 33 2022-11-23T03:20:05.3395847Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55892 2022-11-23T03:20:05.3396317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55893 2022-11-23T03:20:05.3396953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3397421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3398042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3398531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3398991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3399511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3400271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3400785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3401417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3401908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3402366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3402884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3403598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3404351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3404762Z ok (4.235s) 2022-11-23T03:20:05.3405350Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 32 2022-11-23T03:20:05.3405908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56023 2022-11-23T03:20:05.3406379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56024 2022-11-23T03:20:05.3407039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3407512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3408135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3408625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3409070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3409591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3410268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3410736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3411359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3411850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3412306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3412805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3413571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3414326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3414738Z ok (3.734s) 2022-11-23T03:20:05.3415344Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 31 2022-11-23T03:20:05.3415896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56154 2022-11-23T03:20:05.3416368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56155 2022-11-23T03:20:05.3417023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3417477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3418102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3418600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3419096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3419613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3420297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3420761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3421365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3421856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3422321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3422834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3423552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3424300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3424711Z ok (3.734s) 2022-11-23T03:20:05.3425293Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 30 2022-11-23T03:20:05.3425846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56285 2022-11-23T03:20:05.3426315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56286 2022-11-23T03:20:05.3426962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3427433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3428050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3428541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3428999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3429498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3430173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3430641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3431258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3431796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3432253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3432771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3433476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3434227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3434633Z ok (4.135s) 2022-11-23T03:20:05.3435239Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 29 2022-11-23T03:20:05.3435796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56416 2022-11-23T03:20:05.3436274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56417 2022-11-23T03:20:05.3436929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3437385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3438053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3438552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3439011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3439524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3440291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3440770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3441403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3441878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3442340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3442851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3443568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3444315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3444719Z ok (3.835s) 2022-11-23T03:20:05.3445317Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 28 2022-11-23T03:20:05.3445858Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56547 2022-11-23T03:20:05.3446326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56548 2022-11-23T03:20:05.3446984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3447453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3448075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3448563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3449028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3449527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3450272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3450743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3451365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3451856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3452312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3452830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3453548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3454277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3454687Z ok (4.035s) 2022-11-23T03:20:05.3455285Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 27 2022-11-23T03:20:05.3455839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56678 2022-11-23T03:20:05.3456358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56679 2022-11-23T03:20:05.3457018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3457486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3458092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3458580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3459037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3459552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3460228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3460700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3461323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3461813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3462410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3462920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3463627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3464380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3464790Z ok (4.535s) 2022-11-23T03:20:05.3465400Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 26 2022-11-23T03:20:05.3465949Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56809 2022-11-23T03:20:05.3466405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56810 2022-11-23T03:20:05.3467056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3467524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3468148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3468644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3469153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3469665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3470338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3470811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3471436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3471932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3472393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3472912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3473634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3474382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3474781Z ok (4.036s) 2022-11-23T03:20:05.3475431Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 25 2022-11-23T03:20:05.3475988Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56940 2022-11-23T03:20:05.3476461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56941 2022-11-23T03:20:05.3477123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3477595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3478224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3478704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3479166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3479685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3480463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3480944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3481574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3482070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3482516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3483036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3483750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3484508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3484918Z ok (3.935s) 2022-11-23T03:20:05.3485521Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 24 2022-11-23T03:20:05.3486079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57071 2022-11-23T03:20:05.3486546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57072 2022-11-23T03:20:05.3487190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3487752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3488378Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3488874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3489330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3489847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3490529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3490984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3491607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3492105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3492564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3493081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3493854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3494610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3495015Z ok (4.133s) 2022-11-23T03:20:05.3495606Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 23 2022-11-23T03:20:05.3496163Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57202 2022-11-23T03:20:05.3496628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57203 2022-11-23T03:20:05.3497283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3497752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3498374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3498874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3499317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3499830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3500517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3500987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3501610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3502101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3502558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3503056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3503773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3504518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3504925Z ok (3.832s) 2022-11-23T03:20:05.3505529Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 22 2022-11-23T03:20:05.3506132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57333 2022-11-23T03:20:05.3506603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57334 2022-11-23T03:20:05.3507256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3507715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3508335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3508825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3509283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3509797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3510472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3510940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3511548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3512083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3512536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3513049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3513759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3514507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3514917Z ok (3.834s) 2022-11-23T03:20:05.3515507Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 21 2022-11-23T03:20:05.3516061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 420 2022-11-23T03:20:05.3516523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 421 2022-11-23T03:20:05.3517176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3517651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3518276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3518772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3519230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3519732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3520475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3520945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3521575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3522072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3522537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3523056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3523758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3524505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3524975Z ok (3.835s) 2022-11-23T03:20:05.3525581Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 20 2022-11-23T03:20:05.3526131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 551 2022-11-23T03:20:05.3526596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 552 2022-11-23T03:20:05.3527243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3527695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3528317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3528811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3529276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3529790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3530523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3530991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3531618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3532090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3532546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3533060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3533773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3534524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3534934Z ok (3.935s) 2022-11-23T03:20:05.3535538Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 19 2022-11-23T03:20:05.3536071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 682 2022-11-23T03:20:05.3536533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 683 2022-11-23T03:20:05.3537180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3537648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3538269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3538767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3539222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3554818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3555629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3556117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3556756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3557254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3557700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3558391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3559120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3559872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3560344Z ok (3.836s) 2022-11-23T03:20:05.3560944Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 18 2022-11-23T03:20:05.3561482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 814 2022-11-23T03:20:05.3561937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 815 2022-11-23T03:20:05.3562586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3563047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3563664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3564156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3564865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3565525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3565981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3566594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3567075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3567521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3568033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3568541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3569248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3569988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3570378Z ok (3.935s) 2022-11-23T03:20:05.3570968Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 17 2022-11-23T03:20:05.3571506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 947 2022-11-23T03:20:05.3571965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 948 2022-11-23T03:20:05.3572608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3573073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3573683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3574160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3574604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3575109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3575776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3576233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3576841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3577377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3577812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3578309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3579011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3579756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3580148Z ok (3.835s) 2022-11-23T03:20:05.3580741Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 16 2022-11-23T03:20:05.3581287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1078 2022-11-23T03:20:05.3581756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1079 2022-11-23T03:20:05.3582389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3582849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3583510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3583994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3584440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3585099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3585559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3586166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3586645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3587086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3587598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3588113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3588818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3589549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3589944Z ok (3.735s) 2022-11-23T03:20:05.3590526Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 15 2022-11-23T03:20:05.3591072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1209 2022-11-23T03:20:05.3591533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1210 2022-11-23T03:20:05.3592177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3592638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3593252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3593736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3594177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3594691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3595417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3595884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3596507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3596997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3597457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3597959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3598672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3599412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3599822Z ok (3.835s) 2022-11-23T03:20:05.3600471Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 14 2022-11-23T03:20:05.3601019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1340 2022-11-23T03:20:05.3601543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1341 2022-11-23T03:20:05.3602196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3602650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3603278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3603770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3604295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3604812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3605487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3605955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3606564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3607057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3607513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3608024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3608732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3609474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3609881Z ok (3.733s) 2022-11-23T03:20:05.3610468Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 13 2022-11-23T03:20:05.3611020Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1471 2022-11-23T03:20:05.3611489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1472 2022-11-23T03:20:05.3612135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3612603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3613224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3613715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3614232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3614733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3615413Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3615881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3616500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3616986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3617438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3617954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3618658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3619403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3619810Z ok (4.140s) 2022-11-23T03:20:05.3620450Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 12 2022-11-23T03:20:05.3620999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1602 2022-11-23T03:20:05.3621465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1603 2022-11-23T03:20:05.3622123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3622577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3623199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3623695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3624149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3624666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3625340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3625804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3626429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3626913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3627372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3627885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3628596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3629346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3629753Z ok (4.235s) 2022-11-23T03:20:05.3630350Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 11 2022-11-23T03:20:05.3630889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1733 2022-11-23T03:20:05.3631359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1734 2022-11-23T03:20:05.3632005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3632521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3633145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3633628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3634095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3634595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3635274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3635743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3636364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3636852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3637304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3637809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3638599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3639340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3639740Z ok (4.134s) 2022-11-23T03:20:05.3640407Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 10 2022-11-23T03:20:05.3640958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1864 2022-11-23T03:20:05.3641423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1865 2022-11-23T03:20:05.3642076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3642544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3643148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3643642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3644101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3644613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3645288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3645752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3646376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3646864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3647308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3647823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3648533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3649275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3649674Z ok (4.135s) 2022-11-23T03:20:05.3650268Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 9 2022-11-23T03:20:05.3650874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1995 2022-11-23T03:20:05.3651327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1996 2022-11-23T03:20:05.3651970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3652438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3653064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3653552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3654011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3654525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3655192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3655670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3656289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3656824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3657283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3657794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3658514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3659256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3659643Z ok (4.234s) 2022-11-23T03:20:05.3660242Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 8 2022-11-23T03:20:05.3660789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2126 2022-11-23T03:20:05.3661257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2127 2022-11-23T03:20:05.3661904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3662372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3662996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3663475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3663935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3664448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3665285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3665750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3666378Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3666869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3667314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3667825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3668533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3669281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3669738Z ok (4.336s) 2022-11-23T03:20:05.3670336Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 7 2022-11-23T03:20:05.3670877Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2257 2022-11-23T03:20:05.3671354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2258 2022-11-23T03:20:05.3671990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3672453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3673074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3673563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3674016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3674531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3675207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3675706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3676332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3676821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3677281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3677793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3678499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3679253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3679665Z ok (4.336s) 2022-11-23T03:20:05.3680302Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 6 2022-11-23T03:20:05.3680845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2388 2022-11-23T03:20:05.3681313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2389 2022-11-23T03:20:05.3681968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3682494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3683115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3683616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3684058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3684576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3685252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3685715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3686338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3686828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3687286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3687786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3688554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3689300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3689702Z ok (4.034s) 2022-11-23T03:20:05.3690294Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 5 2022-11-23T03:20:05.3690827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2519 2022-11-23T03:20:05.3691291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2520 2022-11-23T03:20:05.3691929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3692394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3693017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3693502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3694007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3694525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3695198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3695665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3696271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3696766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3697222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3697732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3698445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3699183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3699584Z ok (4.437s) 2022-11-23T03:20:05.3700169Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 4 2022-11-23T03:20:05.3700716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2650 2022-11-23T03:20:05.3701187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2651 2022-11-23T03:20:05.3701835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3702304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3702925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3703419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3703863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3704380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3705061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3705521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3706150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3706687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3707152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3707676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3708377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3709133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3709535Z ok (4.235s) 2022-11-23T03:20:05.3710135Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 3 2022-11-23T03:20:05.3710697Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2781 2022-11-23T03:20:05.3711169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2782 2022-11-23T03:20:05.3711823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3712285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3712936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3713430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3713888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3714405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3715096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3715557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3716179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3716660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3717132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3717636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3718354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3719099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3719510Z ok (4.534s) 2022-11-23T03:20:05.3720104Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 2 2022-11-23T03:20:05.3720711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2912 2022-11-23T03:20:05.3721165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2913 2022-11-23T03:20:05.3721814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3722277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3722908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3723407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3723872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3724384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3725106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3725572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3726192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3726682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3727138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3727645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3728361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3729104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3729542Z ok (4.335s) 2022-11-23T03:20:05.3730164Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 1 2022-11-23T03:20:05.3730723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3043 2022-11-23T03:20:05.3731251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3044 2022-11-23T03:20:05.3731912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3732379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3733004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3733506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3733953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3734472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3735139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3735603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3736221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3736723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3737186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3737705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3738414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3739178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3739571Z ok (4.237s) 2022-11-23T03:20:05.3740223Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_rowwise_to_colwise succeeded - num_retries_left: 0 2022-11-23T03:20:05.3740789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3174 2022-11-23T03:20:05.3741259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3175 2022-11-23T03:20:05.3741913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3742385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3743010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3743504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3744002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3744516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3745211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3745685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3746305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3746801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3747252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3747754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3748470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3749210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3749617Z ok (4.840s) 2022-11-23T03:20:05.3750300Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 50 2022-11-23T03:20:05.3750879Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3305 2022-11-23T03:20:05.3751346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3306 2022-11-23T03:20:05.3751998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3752456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3753077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3753571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3754026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3754540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3755220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3755686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3756292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3756784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3757240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3757757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3758474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3759228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3759637Z ok (4.839s) 2022-11-23T03:20:05.3760313Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 49 2022-11-23T03:20:05.3760892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3436 2022-11-23T03:20:05.3761370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3437 2022-11-23T03:20:05.3762022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3762550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3763174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3763669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3764119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3764783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3765467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3765937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3766557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3767090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3767567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3768085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3768842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3769590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3769996Z ok (4.941s) 2022-11-23T03:20:05.3770638Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 48 2022-11-23T03:20:05.3771228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3567 2022-11-23T03:20:05.3771725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3568 2022-11-23T03:20:05.3775014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3775498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3776150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3776649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3777094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3777613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3778303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3778778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3779405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3779906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3780371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3780874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3781590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3782342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3782749Z ok (4.840s) 2022-11-23T03:20:05.3783384Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 47 2022-11-23T03:20:05.3784075Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3698 2022-11-23T03:20:05.3784547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3699 2022-11-23T03:20:05.3785208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3785665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3786299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3786792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3787253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3787769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3788449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3788921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3789534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3790082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3790549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3791064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3791778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3792523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3792926Z ok (4.539s) 2022-11-23T03:20:05.3793562Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 46 2022-11-23T03:20:05.3794124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3829 2022-11-23T03:20:05.3794599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3830 2022-11-23T03:20:05.3795255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3795724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3796345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3796831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3797352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3797871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3798575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3799043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3799684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3800244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3800710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3801233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3801947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3802829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3803222Z ok (4.739s) 2022-11-23T03:20:05.3803857Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 45 2022-11-23T03:20:05.3804429Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3960 2022-11-23T03:20:05.3804895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3961 2022-11-23T03:20:05.3805554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3806025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3806656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3807135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3807593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3808105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3808833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3809301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3809928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3810422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3810888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3811387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3812098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3812838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3813245Z ok (5.141s) 2022-11-23T03:20:05.3813884Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 44 2022-11-23T03:20:05.3814455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4091 2022-11-23T03:20:05.3814924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4092 2022-11-23T03:20:05.3815561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3816035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3816653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3817146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3817615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3818137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3818819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3819286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3819900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3820415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3820902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3821510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3822232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3822996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3823398Z ok (4.539s) 2022-11-23T03:20:05.3824076Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 43 2022-11-23T03:20:05.3824665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4222 2022-11-23T03:20:05.3825146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4223 2022-11-23T03:20:05.3825777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3826244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3826866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3827440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3827923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3828434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3829147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3829617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3830229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3830727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3831189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3831710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3832430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3833179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3833584Z ok (4.840s) 2022-11-23T03:20:05.3834228Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 42 2022-11-23T03:20:05.3834788Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4353 2022-11-23T03:20:05.3835258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4354 2022-11-23T03:20:05.3835909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3836383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3837011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3837504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3837964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3838463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3839141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3839608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3840369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3840870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3841351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3841869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3842588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3843316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3843720Z ok (4.539s) 2022-11-23T03:20:05.3844357Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 41 2022-11-23T03:20:05.3844934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4484 2022-11-23T03:20:05.3845420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4485 2022-11-23T03:20:05.3846146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3846619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3847233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3847722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3848183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3848711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3849399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3849866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3850491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3850988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3851437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3851948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3852664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3853418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3853833Z ok (4.638s) 2022-11-23T03:20:05.3854473Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 40 2022-11-23T03:20:05.3855042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4615 2022-11-23T03:20:05.3855501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4616 2022-11-23T03:20:05.3856141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3856614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3857240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3857730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3858187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3858767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3859442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3859901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3860519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3861019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3861478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3862155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3862872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3863626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3864023Z ok (4.636s) 2022-11-23T03:20:05.3864661Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 39 2022-11-23T03:20:05.3865305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4746 2022-11-23T03:20:05.3865777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4747 2022-11-23T03:20:05.3866447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3866921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3867543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3868022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3868477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3869145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3869622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3870243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3870740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3871203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3871706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3872232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3872952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3873702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3874114Z ok (4.738s) 2022-11-23T03:20:05.3874754Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 38 2022-11-23T03:20:05.3875323Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4877 2022-11-23T03:20:05.3875795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4878 2022-11-23T03:20:05.3876430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3876898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3877594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3878092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3878560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3879080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3879758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3880269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3880895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3881388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3881850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3882371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3883093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3883901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3884311Z ok (4.940s) 2022-11-23T03:20:05.3884935Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 37 2022-11-23T03:20:05.3885517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5008 2022-11-23T03:20:05.3885993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5009 2022-11-23T03:20:05.3886648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3887126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3887760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3888265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3888717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3889239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3889928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3890405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3891035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3891535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3892010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3892517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3893251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3894007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3894426Z ok (4.941s) 2022-11-23T03:20:05.3895073Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 36 2022-11-23T03:20:05.3895653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5139 2022-11-23T03:20:05.3896127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5140 2022-11-23T03:20:05.3896856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3897321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3897952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3898453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3898919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3899437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3900124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3900600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3901212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3914300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3915039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3915571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3916379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3917135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3917552Z ok (5.041s) 2022-11-23T03:20:05.3918203Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 35 2022-11-23T03:20:05.3918775Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5270 2022-11-23T03:20:05.3919249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5271 2022-11-23T03:20:05.3919914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3920464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3921089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3921586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3922052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3922556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3923243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3923714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3924333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3924824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3925288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3925807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3926513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3927263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3927754Z ok (4.838s) 2022-11-23T03:20:05.3928398Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 34 2022-11-23T03:20:05.3928971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5401 2022-11-23T03:20:05.3929438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5402 2022-11-23T03:20:05.3930093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3930562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3931172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3931663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3932115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3932636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3933311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3933780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3934458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3934935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3935397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3935906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3936622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3937364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3937768Z ok (4.436s) 2022-11-23T03:20:05.3938399Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 33 2022-11-23T03:20:05.3938979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5532 2022-11-23T03:20:05.3939434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5533 2022-11-23T03:20:05.3940078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3940547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3941165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3941653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3942116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3942635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3943305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3943775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3944395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3944878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3945325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3945839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3946612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3947352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3947741Z ok (4.739s) 2022-11-23T03:20:05.3948379Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 32 2022-11-23T03:20:05.3948949Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5663 2022-11-23T03:20:05.3949419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5664 2022-11-23T03:20:05.3950076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3950545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3951171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3951648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3952100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3952666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3953347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3953815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3954440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3954934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3955382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3955897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3956607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3957352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3957757Z ok (4.338s) 2022-11-23T03:20:05.3958387Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 31 2022-11-23T03:20:05.3958959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5794 2022-11-23T03:20:05.3959432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5795 2022-11-23T03:20:05.3960069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3960599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3961219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3961709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3962165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3962683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3963369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3963827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3964452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3964997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3965453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3965965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3966687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3967435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3967838Z ok (4.740s) 2022-11-23T03:20:05.3968474Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 30 2022-11-23T03:20:05.3969049Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5925 2022-11-23T03:20:05.3969518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5926 2022-11-23T03:20:05.3970164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3970634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3971307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3971800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3972243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3972757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3973438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3973903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3974528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3975021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3975479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3975989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3976701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3977443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3977856Z ok (4.739s) 2022-11-23T03:20:05.3978489Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 29 2022-11-23T03:20:05.3979073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6056 2022-11-23T03:20:05.3979550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6057 2022-11-23T03:20:05.3980208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3980667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3981289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3981780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3982239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3982756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3983438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3983959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3984575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3985064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3985524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3986036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3986759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3987507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3987912Z ok (4.337s) 2022-11-23T03:20:05.3988537Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 28 2022-11-23T03:20:05.3989099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6187 2022-11-23T03:20:05.3989620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6188 2022-11-23T03:20:05.3990278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3990758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3991376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3991870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3992336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.3992840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.3993522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.3993988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.3994607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.3995090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.3995549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.3996065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.3996765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3997521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.3997926Z ok (4.538s) 2022-11-23T03:20:05.3998557Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 27 2022-11-23T03:20:05.3999128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6318 2022-11-23T03:20:05.3999604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6319 2022-11-23T03:20:05.4000337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4000815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4001432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4001930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4002451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4002958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4003640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4004109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4004730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4005204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4005666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4006177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4006887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4007638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4008048Z ok (4.938s) 2022-11-23T03:20:05.4008739Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 26 2022-11-23T03:20:05.4009302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6449 2022-11-23T03:20:05.4009774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6450 2022-11-23T03:20:05.4010421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4010892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4011514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4012009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4012466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4012988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4013653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4014121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4014739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4015232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4015694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4016212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4016929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4017668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4018083Z ok (4.639s) 2022-11-23T03:20:05.4018722Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 25 2022-11-23T03:20:05.4019286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6580 2022-11-23T03:20:05.4019762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6581 2022-11-23T03:20:05.4020419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4020943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4021559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4022062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4022587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4023098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4023778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4024250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4024873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4025373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4025821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4026330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4027088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4027842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4028243Z ok (4.337s) 2022-11-23T03:20:05.4028877Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 24 2022-11-23T03:20:05.4029447Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6711 2022-11-23T03:20:05.4029903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6712 2022-11-23T03:20:05.4030558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4031031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4031660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4032149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4032616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4033132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4033813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4034263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4034885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4035371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4035829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4036340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4037056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4037800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4038194Z ok (4.740s) 2022-11-23T03:20:05.4038828Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 23 2022-11-23T03:20:05.4039458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6842 2022-11-23T03:20:05.4039924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6843 2022-11-23T03:20:05.4040658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4041134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4041763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4042242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4042704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4043373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4043844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4044461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4044955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4045477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4045996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4046509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4047227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4047973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4048391Z ok (4.940s) 2022-11-23T03:20:05.4049026Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 22 2022-11-23T03:20:05.4049603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6973 2022-11-23T03:20:05.4050078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6974 2022-11-23T03:20:05.4050718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4051187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4051800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4052292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4052752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4053271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4053961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4054423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4055205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4055690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4056149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4056664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4057382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4058219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4058633Z ok (4.740s) 2022-11-23T03:20:05.4059255Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 21 2022-11-23T03:20:05.4059829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7104 2022-11-23T03:20:05.4060298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7105 2022-11-23T03:20:05.4060955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4061429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4062054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4062543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4062986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4063499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4064230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4064704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4065324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4065809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4066275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4066794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4067507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4068269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4068676Z ok (4.839s) 2022-11-23T03:20:05.4069326Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 20 2022-11-23T03:20:05.4069899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7235 2022-11-23T03:20:05.4070368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7236 2022-11-23T03:20:05.4071020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4071480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4072102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4072596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4073053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4073568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4074245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4074717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4075338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4075819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4076275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4076968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4077696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4078462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4078870Z ok (4.438s) 2022-11-23T03:20:05.4079501Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 19 2022-11-23T03:20:05.4080059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7366 2022-11-23T03:20:05.4080600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7367 2022-11-23T03:20:05.4081255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4081736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4082362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4082916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4083377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4084031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4084500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4085115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4085600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4086060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4086584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4087121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4087837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4088573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4088981Z ok (4.636s) 2022-11-23T03:20:05.4089612Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 18 2022-11-23T03:20:05.4090179Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7497 2022-11-23T03:20:05.4090652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7498 2022-11-23T03:20:05.4091310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4091778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4092389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4092886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4093350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4093869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4094553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4094731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4095200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4095392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4095629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4095881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4096316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4096747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4096842Z ok (4.839s) 2022-11-23T03:20:05.4097338Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 17 2022-11-23T03:20:05.4097549Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7628 2022-11-23T03:20:05.4097769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7629 2022-11-23T03:20:05.4098219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4098401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4098816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4099009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4099239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4099502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4099902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4100078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4100484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4100681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4100902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4101150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4101578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4102006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4102105Z ok (4.838s) 2022-11-23T03:20:05.4102600Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 16 2022-11-23T03:20:05.4102824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7759 2022-11-23T03:20:05.4103046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7760 2022-11-23T03:20:05.4103451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4103627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4104040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4104237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4104472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4104778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4105176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4105354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4105764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4105953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4106183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4106435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4106865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4107299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4107379Z ok (4.841s) 2022-11-23T03:20:05.4107875Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 15 2022-11-23T03:20:05.4108145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7890 2022-11-23T03:20:05.4108363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7891 2022-11-23T03:20:05.4108760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4108934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4109342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4109532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4109767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4110017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4110418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4110593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4111008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4111199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4111433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4111681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4112125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4112555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4112646Z ok (4.841s) 2022-11-23T03:20:05.4113146Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 14 2022-11-23T03:20:05.4113367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8021 2022-11-23T03:20:05.4113586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8022 2022-11-23T03:20:05.4113969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4114144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4114608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4114802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4115038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4115290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4115690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4115869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4116284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4116478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4116717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4116974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4117411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4117886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4117985Z ok (4.639s) 2022-11-23T03:20:05.4118487Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 13 2022-11-23T03:20:05.4118711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8152 2022-11-23T03:20:05.4118930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8153 2022-11-23T03:20:05.4119333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4119515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4119927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4120125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4120424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4120682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4121087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4121261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4121670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4121870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4122107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4122356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4122793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4123223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4123316Z ok (4.739s) 2022-11-23T03:20:05.4123811Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 12 2022-11-23T03:20:05.4124037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8283 2022-11-23T03:20:05.4124312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8284 2022-11-23T03:20:05.4124717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4124896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4125317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4125511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4125747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4125997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4126399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4126575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4126973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4127173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4127454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4127709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4128144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4128577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4128676Z ok (4.839s) 2022-11-23T03:20:05.4129173Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 11 2022-11-23T03:20:05.4129402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8414 2022-11-23T03:20:05.4129625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8415 2022-11-23T03:20:05.4130029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4130206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4130620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4130817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4131054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4131307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4131707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4131885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4132299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4132495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4132734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4132988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4133407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4133835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4133986Z ok (4.739s) 2022-11-23T03:20:05.4134484Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 10 2022-11-23T03:20:05.4134703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8545 2022-11-23T03:20:05.4134923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8546 2022-11-23T03:20:05.4135324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4135506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4135921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4136110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4136344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4136594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4136992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4137171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4137626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4137819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4138053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4138298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4138738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4139174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4139266Z ok (4.637s) 2022-11-23T03:20:05.4139759Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 9 2022-11-23T03:20:05.4139966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8676 2022-11-23T03:20:05.4140183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8677 2022-11-23T03:20:05.4140585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4140760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4141172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4141365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4141600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4141850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4142246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4142426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4142836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4143030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4143265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4143513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4144002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4144429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4144524Z ok (4.538s) 2022-11-23T03:20:05.4145018Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 8 2022-11-23T03:20:05.4145241Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8807 2022-11-23T03:20:05.4145458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8808 2022-11-23T03:20:05.4145864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4146043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4146440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4146631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4146861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4147171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4147571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4147747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4148151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4148342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4148576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4148828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4149259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4149696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4149793Z ok (4.838s) 2022-11-23T03:20:05.4150283Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 7 2022-11-23T03:20:05.4150502Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8938 2022-11-23T03:20:05.4150722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8939 2022-11-23T03:20:05.4151121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4151300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4151708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4151904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4152137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4152383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4152764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4152939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4153349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4153751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4153987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4154236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4154680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4155113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4155207Z ok (4.940s) 2022-11-23T03:20:05.4155693Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 6 2022-11-23T03:20:05.4155919Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9069 2022-11-23T03:20:05.4156142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9070 2022-11-23T03:20:05.4156542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4156714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4157164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4157357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4157592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4157844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4158251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4158428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4158845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4159038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4159262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4159512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4159948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4160449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4160546Z ok (5.041s) 2022-11-23T03:20:05.4161042Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 5 2022-11-23T03:20:05.4161270Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9200 2022-11-23T03:20:05.4161493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9201 2022-11-23T03:20:05.4161894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4162068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4162482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4162682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4162918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4163167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4163565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4163807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4164219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4164415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4164645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4164893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4165323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4165750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4165834Z ok (4.740s) 2022-11-23T03:20:05.4166319Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 4 2022-11-23T03:20:05.4166540Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9331 2022-11-23T03:20:05.4166807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9332 2022-11-23T03:20:05.4167212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4167390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4167805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4167998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4168232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4168483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4168878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4169051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4169461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4169656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4169888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4170139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4170575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4171015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4171111Z ok (4.438s) 2022-11-23T03:20:05.4171599Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 3 2022-11-23T03:20:05.4171824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9462 2022-11-23T03:20:05.4172040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9463 2022-11-23T03:20:05.4172424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4172603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4173009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4173204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4173489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4173740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4174144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4174323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4174737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4174925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4175154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4175403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4175840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4176266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4176362Z ok (4.641s) 2022-11-23T03:20:05.4176901Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 2 2022-11-23T03:20:05.4177125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9593 2022-11-23T03:20:05.4177344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9594 2022-11-23T03:20:05.4177748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4177924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4178341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4178533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4178750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4179004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4179404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4179580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4179992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4180186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4180422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4180671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4181103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4181532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4181626Z ok (4.737s) 2022-11-23T03:20:05.4182116Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 1 2022-11-23T03:20:05.4182340Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9724 2022-11-23T03:20:05.4182559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9725 2022-11-23T03:20:05.4182958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4183181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4183597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4183791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4184025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:20:05.4184275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:20:05.4184671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:20:05.4184848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:20:05.4185241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:20:05.4185436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:20:05.4185671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:20:05.4185921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:20:05.4186396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4186832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:20:05.4186928Z ok (4.838s) 2022-11-23T03:20:05.4187393Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... test_load_with_different_shard_plan succeeded - num_retries_left: 0 2022-11-23T03:20:05.4187802Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T03:20:05.4188435Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.006s) 2022-11-23T03:20:05.4189042Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:05.4189722Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:05.4189735Z 2022-11-23T03:20:05.4190024Z ---------------------------------------------------------------------- 2022-11-23T03:20:05.4190139Z Ran 106 tests in 447.118s 2022-11-23T03:20:05.4190146Z 2022-11-23T03:20:05.4190247Z OK (skipped=4) 2022-11-23T03:20:05.4190254Z 2022-11-23T03:20:05.4190374Z Generating XML reports... 2022-11-23T03:20:05.4190960Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123031236.xml 2022-11-23T03:20:05.4191559Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123031236.xml 2022-11-23T03:20:05.4192248Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123031236.xml 2022-11-23T03:20:05.4192257Z 2022-11-23T03:20:05.4192721Z ##[endgroup] 2022-11-23T03:20:05.4193322Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_2ooude1n) 2022-11-23T03:20:05.4193330Z 2022-11-23T03:20:05.4193639Z Running distributed/checkpoint/test_checkpoint ... [2022-11-23 03:20:05.319330] 2022-11-23T03:20:05.4194311Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:05.320204] 2022-11-23T03:20:08.8695471Z 2022-11-23T03:20:08.8696604Z Expand the folded group to see the log file of distributed/checkpoint/test_checkpoint 2022-11-23T03:20:08.8699361Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_checkpoint_ws14vqxe) 2022-11-23T03:20:08.8701704Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_checkpoint 2022-11-23T03:20:08.8702495Z 2022-11-23T03:20:08.8702723Z Running tests... 2022-11-23T03:20:08.8703884Z ---------------------------------------------------------------------- 2022-11-23T03:20:08.8706025Z test_default_metadata (__main__.TestDistributedCheckpointing) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.530s) 2022-11-23T03:20:08.8708718Z test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:08.8711755Z test_dummy_reader_works (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:08.8714209Z test_dummy_writer_works (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:08.8716620Z test_load_error_handling (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:08.8719060Z test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:08.8721670Z test_save_error_handling (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:08.8724114Z test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:08.8725123Z 2022-11-23T03:20:08.8725881Z ---------------------------------------------------------------------- 2022-11-23T03:20:08.8726737Z Ran 8 tests in 0.539s 2022-11-23T03:20:08.8727148Z 2022-11-23T03:20:08.8727399Z OK (skipped=8) 2022-11-23T03:20:08.8727784Z 2022-11-23T03:20:08.8728084Z Generating XML reports... 2022-11-23T03:20:08.8729969Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123032006.xml 2022-11-23T03:20:08.8732411Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123032006.xml 2022-11-23T03:20:08.8733442Z 2022-11-23T03:20:08.8734231Z ##[endgroup] 2022-11-23T03:20:08.8736012Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_checkpoint_ws14vqxe) 2022-11-23T03:20:08.8737045Z 2022-11-23T03:20:08.8737941Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2022-11-23 03:20:08.870214] 2022-11-23T03:20:08.8740192Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:08.871075] 2022-11-23T03:20:12.3501402Z 2022-11-23T03:20:12.3502674Z Expand the folded group to see the log file of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2022-11-23T03:20:12.3505714Z ##[group]PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_ekhuy4r5) 2022-11-23T03:20:12.3508177Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks 2022-11-23T03:20:12.3509010Z 2022-11-23T03:20:12.3509280Z Running tests... 2022-11-23T03:20:12.3510425Z ---------------------------------------------------------------------- 2022-11-23T03:20:12.3512130Z test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:20:12.3514428Z This unit test verifies the ``allreduce`` hook registered case gives same result ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.488s) 2022-11-23T03:20:12.3516124Z test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:20:12.3518465Z This unit test verifies the ``fp16 compress`` hook registered case ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:12.3520654Z test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:20:12.3523198Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:12.3524937Z test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:20:12.3527182Z This unit test verifies the ``quantize per channel`` hook registered case ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:12.3528868Z test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:20:12.3531054Z This unit test verifies the ``quantize per tensor`` hook registered case ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:12.3533584Z test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:12.3534662Z 2022-11-23T03:20:12.3535401Z ---------------------------------------------------------------------- 2022-11-23T03:20:12.3536266Z Ran 6 tests in 0.492s 2022-11-23T03:20:12.3536669Z 2022-11-23T03:20:12.3536917Z OK (skipped=6) 2022-11-23T03:20:12.3537286Z 2022-11-23T03:20:12.3537575Z Generating XML reports... 2022-11-23T03:20:12.3539583Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20221123032010.xml 2022-11-23T03:20:12.3540796Z 2022-11-23T03:20:12.3541597Z ##[endgroup] 2022-11-23T03:20:12.3543467Z FINISHED PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_ekhuy4r5) 2022-11-23T03:20:12.3544561Z 2022-11-23T03:20:12.3545370Z Running distributed/_tensor/test_tp_sharding_ops ... [2022-11-23 03:20:12.350821] 2022-11-23T03:20:12.3547520Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tp_sharding_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:12.351676] 2022-11-23T03:20:15.9089032Z 2022-11-23T03:20:15.9090851Z Expand the folded group to see the log file of distributed/_tensor/test_tp_sharding_ops 2022-11-23T03:20:15.9094616Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_tp_sharding_ops_5snjzntl) 2022-11-23T03:20:15.9096900Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops 2022-11-23T03:20:15.9098301Z 2022-11-23T03:20:15.9098555Z Running tests... 2022-11-23T03:20:15.9099702Z ---------------------------------------------------------------------- 2022-11-23T03:20:15.9102957Z test_replicated_permute (__main__.TPShardingOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.539s) 2022-11-23T03:20:15.9105341Z test_sharded_cat (__main__.TPShardingOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:15.9108184Z test_sharded_permute (__main__.TPShardingOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:15.9111191Z test_sharded_split (__main__.TPShardingOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:15.9113510Z test_sharded_transpose (__main__.TPShardingOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:15.9115805Z test_sharded_view (__main__.TPShardingOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:15.9116765Z 2022-11-23T03:20:15.9117725Z ---------------------------------------------------------------------- 2022-11-23T03:20:15.9118600Z Ran 6 tests in 0.547s 2022-11-23T03:20:15.9119004Z 2022-11-23T03:20:15.9119251Z OK (skipped=6) 2022-11-23T03:20:15.9119626Z 2022-11-23T03:20:15.9119918Z Generating XML reports... 2022-11-23T03:20:15.9121787Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20221123032013.xml 2022-11-23T03:20:15.9122746Z 2022-11-23T03:20:15.9123518Z ##[endgroup] 2022-11-23T03:20:15.9125222Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_tp_sharding_ops_5snjzntl) 2022-11-23T03:20:15.9126198Z 2022-11-23T03:20:15.9126986Z Running distributed/_tensor/test_redistribute ... [2022-11-23 03:20:15.909594] 2022-11-23T03:20:15.9129134Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:15.910448] 2022-11-23T03:20:19.4583549Z 2022-11-23T03:20:19.4584904Z Expand the folded group to see the log file of distributed/_tensor/test_redistribute 2022-11-23T03:20:19.4588433Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_redistribute_c87czlgw) 2022-11-23T03:20:19.4590683Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_redistribute 2022-11-23T03:20:19.4591418Z 2022-11-23T03:20:19.4591667Z Running tests... 2022-11-23T03:20:19.4592810Z ---------------------------------------------------------------------- 2022-11-23T03:20:19.4595680Z test_multi_dim_mesh (__main__.MultiDimRedistributeTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.517s) 2022-11-23T03:20:19.4598413Z test_partial_to_replicate_forward_backward (__main__.RedistributeTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:19.4600974Z test_partial_to_shard (__main__.RedistributeTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:19.4604484Z test_replicate_to_partial (__main__.RedistributeTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:19.4606928Z test_replicate_to_replicate_forward_backward (__main__.RedistributeTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:19.4609879Z test_replicate_to_shard_forward_backward (__main__.RedistributeTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:19.4612334Z test_shard_to_replicate_forward_backward (__main__.RedistributeTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:19.4613344Z 2022-11-23T03:20:19.4614083Z ---------------------------------------------------------------------- 2022-11-23T03:20:19.4614948Z Ran 7 tests in 0.529s 2022-11-23T03:20:19.4615348Z 2022-11-23T03:20:19.4615592Z OK (skipped=7) 2022-11-23T03:20:19.4615971Z 2022-11-23T03:20:19.4616260Z Generating XML reports... 2022-11-23T03:20:19.4618003Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20221123032017.xml 2022-11-23T03:20:19.4620230Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20221123032017.xml 2022-11-23T03:20:19.4621149Z 2022-11-23T03:20:19.4621901Z ##[endgroup] 2022-11-23T03:20:19.4623762Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_redistribute_c87czlgw) 2022-11-23T03:20:19.4624722Z 2022-11-23T03:20:19.4625498Z Running distributed/_tensor/test_matrix_ops ... [2022-11-23 03:20:19.459014] 2022-11-23T03:20:19.4627596Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:19.459876] 2022-11-23T03:20:23.0464708Z 2022-11-23T03:20:23.0466097Z Expand the folded group to see the log file of distributed/_tensor/test_matrix_ops 2022-11-23T03:20:23.0469514Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_matrix_ops_zu2z5p18) 2022-11-23T03:20:23.0471745Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_matrix_ops 2022-11-23T03:20:23.0472474Z 2022-11-23T03:20:23.0472731Z Running tests... 2022-11-23T03:20:23.0473857Z ---------------------------------------------------------------------- 2022-11-23T03:20:23.0475768Z test_addmm (__main__.DistMatrixOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.521s) 2022-11-23T03:20:23.0478108Z test_addmm_auto_redistribute (__main__.DistMatrixOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:23.0480605Z test_baddbmm (__main__.DistMatrixOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T03:20:23.0482819Z test_bmm (__main__.DistMatrixOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:23.0484983Z test_mm (__main__.DistMatrixOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:23.0487141Z test_t (__main__.DistMatrixOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:23.0489296Z test_t_partial (__main__.DistMatrixOpsTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:23.0490219Z 2022-11-23T03:20:23.0490961Z ---------------------------------------------------------------------- 2022-11-23T03:20:23.0491822Z Ran 7 tests in 0.533s 2022-11-23T03:20:23.0492221Z 2022-11-23T03:20:23.0492465Z OK (skipped=7) 2022-11-23T03:20:23.0492843Z 2022-11-23T03:20:23.0493575Z Generating XML reports... 2022-11-23T03:20:23.0495243Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20221123032021.xml 2022-11-23T03:20:23.0496178Z 2022-11-23T03:20:23.0496941Z ##[endgroup] 2022-11-23T03:20:23.0498594Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_matrix_ops_zu2z5p18) 2022-11-23T03:20:23.0499507Z 2022-11-23T03:20:23.0500246Z Running distributed/_tensor/test_dtensor ... [2022-11-23 03:20:23.047099] 2022-11-23T03:20:23.0502302Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:23.047959] 2022-11-23T03:20:26.5688789Z 2022-11-23T03:20:26.5690335Z Expand the folded group to see the log file of distributed/_tensor/test_dtensor 2022-11-23T03:20:26.5694092Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_dtensor_cctsmu5k) 2022-11-23T03:20:26.5696194Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_dtensor 2022-11-23T03:20:26.5696924Z 2022-11-23T03:20:26.5697171Z Running tests... 2022-11-23T03:20:26.5698759Z ---------------------------------------------------------------------- 2022-11-23T03:20:26.5700774Z test_device_mesh_nd (__main__.DTensorMeshTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.480s) 2022-11-23T03:20:26.5703056Z test_dtensor_2d_mesh (__main__.DTensorMeshTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.011s) 2022-11-23T03:20:26.5705437Z test_dtensor_api_device_mesh_context_manager (__main__.DTensorMeshTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s) 2022-11-23T03:20:26.5707933Z test_dtensor_device_mesh_device_conversion (__main__.DTensorMeshTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.005s) 2022-11-23T03:20:26.5710378Z test_dtensor_spec_local_shard_offset (__main__.DTensorMeshTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:26.5712661Z test_dtensor_constructor (__main__.DTensorTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:26.5714868Z test_dtensor_properties (__main__.DTensorTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:26.5717184Z test_dtensor_spec_read_only_after_set (__main__.DTensorTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:26.5719404Z test_dtensor_stride (__main__.DTensorTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:26.5721725Z test_from_local (__main__.DTensorTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:26.5723935Z test_from_local_then_to_local (__main__.DTensorTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:26.5726095Z test_to_local (__main__.DTensorTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:26.5726978Z 2022-11-23T03:20:26.5727713Z ---------------------------------------------------------------------- 2022-11-23T03:20:26.5728573Z Ran 12 tests in 0.512s 2022-11-23T03:20:26.5728978Z 2022-11-23T03:20:26.5729554Z OK (skipped=12) 2022-11-23T03:20:26.5729937Z 2022-11-23T03:20:26.5730223Z Generating XML reports... 2022-11-23T03:20:26.5731853Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20221123032024.xml 2022-11-23T03:20:26.5733861Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20221123032024.xml 2022-11-23T03:20:26.5734714Z 2022-11-23T03:20:26.5735491Z ##[endgroup] 2022-11-23T03:20:26.5737122Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_dtensor_cctsmu5k) 2022-11-23T03:20:26.5738014Z 2022-11-23T03:20:26.5738790Z Running distributed/_tensor/test_common_rules ... [2022-11-23 03:20:26.569348] 2022-11-23T03:20:26.5740883Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:26.570135] 2022-11-23T03:20:30.0845837Z 2022-11-23T03:20:30.0847056Z Expand the folded group to see the log file of distributed/_tensor/test_common_rules 2022-11-23T03:20:30.0850094Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_common_rules_8798jdo0) 2022-11-23T03:20:30.0852578Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_common_rules 2022-11-23T03:20:30.0853270Z 2022-11-23T03:20:30.0853505Z Running tests... 2022-11-23T03:20:30.0854557Z ---------------------------------------------------------------------- 2022-11-23T03:20:30.0856409Z test_einop_basic_propagation (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.494s) 2022-11-23T03:20:30.0858499Z test_einop_errors (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:30.0860555Z test_einop_linearity (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T03:20:30.0862629Z test_einop_merge_sharding (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:30.0864761Z test_einop_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:30.0866911Z test_einop_pointwise_propagation (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:30.0869190Z test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:30.0871428Z test_pointwise_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:30.0873597Z test_pointwise_rules_broadcasting (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:30.0875718Z test_pointwise_rules_suggestion (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:30.0877789Z test_reduction_rule (__main__.CommonRulesTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:30.0878627Z 2022-11-23T03:20:30.0879303Z ---------------------------------------------------------------------- 2022-11-23T03:20:30.0880577Z Ran 11 tests in 0.513s 2022-11-23T03:20:30.0880957Z 2022-11-23T03:20:30.0881189Z OK (skipped=11) 2022-11-23T03:20:30.0881533Z 2022-11-23T03:20:30.0881801Z Generating XML reports... 2022-11-23T03:20:30.0883321Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20221123032028.xml 2022-11-23T03:20:30.0884171Z 2022-11-23T03:20:30.0884870Z ##[endgroup] 2022-11-23T03:20:30.0886602Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_common_rules_8798jdo0) 2022-11-23T03:20:30.0887537Z 2022-11-23T03:20:30.0888458Z Running distributed/_tensor/parallel/test_view_sharding_dim_change ... [2022-11-23 03:20:30.085264] 2022-11-23T03:20:30.0890781Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_view_sharding_dim_change.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:30.086049] 2022-11-23T03:20:33.6220285Z 2022-11-23T03:20:33.6222088Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_view_sharding_dim_change 2022-11-23T03:20:33.6225861Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_view_sharding_dim_change_uuuhq38a) 2022-11-23T03:20:33.6228551Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change 2022-11-23T03:20:33.6229382Z 2022-11-23T03:20:33.6229632Z Running tests... 2022-11-23T03:20:33.6230753Z ---------------------------------------------------------------------- 2022-11-23T03:20:33.6232947Z test_view_with_sharding_dim_change (__main__.TPViewShardingDimChangeTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.511s) 2022-11-23T03:20:33.6234047Z 2022-11-23T03:20:33.6234787Z ---------------------------------------------------------------------- 2022-11-23T03:20:33.6235720Z Ran 1 test in 0.511s 2022-11-23T03:20:33.6236120Z 2022-11-23T03:20:33.6236362Z OK (skipped=1) 2022-11-23T03:20:33.6236737Z 2022-11-23T03:20:33.6237024Z Generating XML reports... 2022-11-23T03:20:33.6238971Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20221123032031.xml 2022-11-23T03:20:33.6240108Z 2022-11-23T03:20:33.6241332Z ##[endgroup] 2022-11-23T03:20:33.6243309Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_view_sharding_dim_change_uuuhq38a) 2022-11-23T03:20:33.6244438Z 2022-11-23T03:20:33.6245287Z Running distributed/_tensor/parallel/test_tp_examples ... [2022-11-23 03:20:33.622588] 2022-11-23T03:20:33.6247495Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_tp_examples.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:33.623443] 2022-11-23T03:20:37.1599333Z 2022-11-23T03:20:37.1601432Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_tp_examples 2022-11-23T03:20:37.1605594Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_examples (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_tp_examples_yex327cx) 2022-11-23T03:20:37.1607983Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_tp_examples 2022-11-23T03:20:37.1608783Z 2022-11-23T03:20:37.1609082Z Running tests... 2022-11-23T03:20:37.1610228Z ---------------------------------------------------------------------- 2022-11-23T03:20:37.1612365Z test_mlp_megatron_e2e (__main__.DistTensorParallelExampleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.531s) 2022-11-23T03:20:37.1615441Z test_self_attn_megatron_e2e (__main__.DistTensorParallelExampleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.005s) 2022-11-23T03:20:37.1618067Z test_self_attn_replacement_megatron_e2e (__main__.DistTensorParallelExampleTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.005s) 2022-11-23T03:20:37.1619188Z 2022-11-23T03:20:37.1619930Z ---------------------------------------------------------------------- 2022-11-23T03:20:37.1620792Z Ran 3 tests in 0.542s 2022-11-23T03:20:37.1621197Z 2022-11-23T03:20:37.1621437Z OK (skipped=3) 2022-11-23T03:20:37.1621818Z 2022-11-23T03:20:37.1622111Z Generating XML reports... 2022-11-23T03:20:37.1624012Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20221123032035.xml 2022-11-23T03:20:37.1625127Z 2022-11-23T03:20:37.1625912Z ##[endgroup] 2022-11-23T03:20:37.1627708Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_examples (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_tp_examples_yex327cx) 2022-11-23T03:20:37.1628742Z 2022-11-23T03:20:37.1629571Z Running distributed/_tensor/parallel/test_2d_parallel ... [2022-11-23 03:20:37.160542] 2022-11-23T03:20:37.1631939Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_2d_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:37.161424] 2022-11-23T03:20:40.7078075Z 2022-11-23T03:20:40.7079466Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_2d_parallel 2022-11-23T03:20:40.7083254Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_mlk_4zj9) 2022-11-23T03:20:40.7085577Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel 2022-11-23T03:20:40.7086399Z 2022-11-23T03:20:40.7086648Z Running tests... 2022-11-23T03:20:40.7087780Z ---------------------------------------------------------------------- 2022-11-23T03:20:40.7089958Z test_2d_fsdp_integration_correctness (__main__.Test2dParallelIntegration) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.566s) 2022-11-23T03:20:40.7092547Z test_2d_fsdp_integration_functionality (__main__.Test2dParallelIntegration) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:40.7093627Z 2022-11-23T03:20:40.7094360Z ---------------------------------------------------------------------- 2022-11-23T03:20:40.7095221Z Ran 2 tests in 0.567s 2022-11-23T03:20:40.7095623Z 2022-11-23T03:20:40.7095863Z OK (skipped=2) 2022-11-23T03:20:40.7096239Z 2022-11-23T03:20:40.7096506Z Generating XML reports... 2022-11-23T03:20:40.7098337Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20221123032038.xml 2022-11-23T03:20:40.7099398Z 2022-11-23T03:20:40.7100143Z ##[endgroup] 2022-11-23T03:20:40.7101958Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_mlk_4zj9) 2022-11-23T03:20:40.7102972Z 2022-11-23T03:20:40.7103849Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-11-23 03:20:40.708401] 2022-11-23T03:20:40.7106096Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:40.709151] 2022-11-23T03:20:44.1765394Z 2022-11-23T03:20:44.1766437Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec 2022-11-23T03:20:44.1771352Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_ujahz1sa) 2022-11-23T03:20:44.1773317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48wmcqaz 2022-11-23T03:20:44.1774802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48wmcqaz/_remote_module_non_scriptable.py 2022-11-23T03:20:44.1776932Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec 2022-11-23T03:20:44.1777715Z 2022-11-23T03:20:44.1777963Z Running tests... 2022-11-23T03:20:44.1779085Z ---------------------------------------------------------------------- 2022-11-23T03:20:44.1781164Z test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.598s) 2022-11-23T03:20:44.1782724Z test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec) 2022-11-23T03:20:44.1784640Z Test custom spec can be invoked from the ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:44.1786063Z test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec) 2022-11-23T03:20:44.1788250Z Test sharded_tensor.ones(...) with the custom ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:44.1790560Z test_check_overlapping (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:44.1792843Z test_chunked_sharding_spec (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:44.1795172Z test_device_placement (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:44.1797535Z test_enumerable_sharding_spec (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s) 2022-11-23T03:20:44.1799915Z test_get_chunk_sharding_params (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:44.1802424Z test_get_chunked_dim_size (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:44.1804707Z test_get_split_size (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:44.1807121Z test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:44.1808175Z 2022-11-23T03:20:44.1808914Z ---------------------------------------------------------------------- 2022-11-23T03:20:44.1809771Z Ran 11 tests in 0.611s 2022-11-23T03:20:44.1810208Z 2022-11-23T03:20:44.1810466Z OK (skipped=11) 2022-11-23T03:20:44.1810848Z 2022-11-23T03:20:44.1811134Z Generating XML reports... 2022-11-23T03:20:44.1812925Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221123032042.xml 2022-11-23T03:20:44.1815276Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221123032042.xml 2022-11-23T03:20:44.1816255Z 2022-11-23T03:20:44.1817030Z ##[endgroup] 2022-11-23T03:20:44.1818906Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_ujahz1sa) 2022-11-23T03:20:44.1820145Z 2022-11-23T03:20:44.1820932Z Running distributed/_composable/test_replicate ... [2022-11-23 03:20:44.177198] 2022-11-23T03:20:44.1823106Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:44.178050] 2022-11-23T03:20:47.7260664Z 2022-11-23T03:20:47.7262027Z Expand the folded group to see the log file of distributed/_composable/test_replicate 2022-11-23T03:20:47.7265490Z ##[group]PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_replicate_qgnsp2uq) 2022-11-23T03:20:47.7267709Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_replicate 2022-11-23T03:20:47.7268469Z 2022-11-23T03:20:47.7268714Z Running tests... 2022-11-23T03:20:47.7269870Z ---------------------------------------------------------------------- 2022-11-23T03:20:47.7271883Z test_replicate_multi_module (__main__.ReplicateTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.488s) 2022-11-23T03:20:47.7274669Z test_replicate_single_module (__main__.ReplicateTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:47.7275631Z 2022-11-23T03:20:47.7276377Z ---------------------------------------------------------------------- 2022-11-23T03:20:47.7277230Z Ran 2 tests in 0.489s 2022-11-23T03:20:47.7277637Z 2022-11-23T03:20:47.7277876Z OK (skipped=2) 2022-11-23T03:20:47.7278237Z 2022-11-23T03:20:47.7278524Z Generating XML reports... 2022-11-23T03:20:47.7280372Z Generated XML report: test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20221123032045.xml 2022-11-23T03:20:47.7281306Z 2022-11-23T03:20:47.7282064Z ##[endgroup] 2022-11-23T03:20:47.7283781Z FINISHED PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_replicate_qgnsp2uq) 2022-11-23T03:20:47.7284740Z 2022-11-23T03:20:47.7285538Z Running distributed/_composable/test_contract ... [2022-11-23 03:20:47.726663] 2022-11-23T03:20:47.7287688Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_contract.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:20:47.727519] 2022-11-23T03:20:51.2296019Z 2022-11-23T03:20:51.2297433Z Expand the folded group to see the log file of distributed/_composable/test_contract 2022-11-23T03:20:51.2300591Z ##[group]PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_contract_x3rthle8) 2022-11-23T03:20:51.2302220Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_contract 2022-11-23T03:20:51.2302775Z 2022-11-23T03:20:51.2302959Z Running tests... 2022-11-23T03:20:51.2303780Z ---------------------------------------------------------------------- 2022-11-23T03:20:51.2305183Z test_add_hooks (__main__.TestContract) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.514s) 2022-11-23T03:20:51.2306746Z test_modify_fqn (__main__.TestContract) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s) 2022-11-23T03:20:51.2308287Z test_state (__main__.TestContract) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s) 2022-11-23T03:20:51.2308931Z 2022-11-23T03:20:51.2309461Z ---------------------------------------------------------------------- 2022-11-23T03:20:51.2310087Z Ran 3 tests in 0.518s 2022-11-23T03:20:51.2310381Z 2022-11-23T03:20:51.2310971Z OK (skipped=3) 2022-11-23T03:20:51.2311246Z 2022-11-23T03:20:51.2311460Z Generating XML reports... 2022-11-23T03:20:51.2312640Z Generated XML report: test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20221123032049.xml 2022-11-23T03:20:51.2313288Z 2022-11-23T03:20:51.2313846Z ##[endgroup] 2022-11-23T03:20:51.2315070Z FINISHED PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_contract_x3rthle8) 2022-11-23T03:20:51.2315756Z 2022-11-23T03:20:52.7567527Z 2022-11-23T03:20:52.7568382Z real 103m44.234s 2022-11-23T03:20:52.7569477Z user 169m40.602s 2022-11-23T03:20:52.7570640Z sys 67m0.015s 2022-11-23T03:20:52.7571620Z + assert_git_not_dirty 2022-11-23T03:20:52.7572952Z + [[ linux-focal-rocm5.2-py3.8 != *rocm* ]] 2022-11-23T03:20:52.7574024Z + [[ linux-focal-rocm5.2-py3.8 == *cuda* ]] 2022-11-23T03:20:52.7574700Z + [[ 2 == 1 ]] 2022-11-23T03:20:52.7726833Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2022-11-23T03:20:52.7727410Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2022-11-23T03:20:52.7728060Z docker exec -t "ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2022-11-23T03:20:52.7757222Z shell: /bin/bash -e {0} 2022-11-23T03:20:52.7757481Z env: 2022-11-23T03:20:52.7757715Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:20:52.7758041Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:20:52.7758557Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:20:52.7759141Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:20:52.7759525Z ##[endgroup] 2022-11-23T03:20:52.9219070Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2022-11-23T03:20:52.9801796Z Prepare all required actions 2022-11-23T03:20:52.9803020Z Getting action download info 2022-11-23T03:20:53.2188344Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2022-11-23T03:20:54.0976249Z ##[group]Run ./.github/actions/get-workflow-job-id 2022-11-23T03:20:54.0976758Z with: 2022-11-23T03:20:54.0978384Z github-token: *** 2022-11-23T03:20:54.0978810Z env: 2022-11-23T03:20:54.0979212Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:20:54.0979724Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:20:54.0980526Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:20:54.0981414Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:20:54.0982010Z ##[endgroup] 2022-11-23T03:20:54.1037020Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2022-11-23T03:20:54.1037541Z with: 2022-11-23T03:20:54.1037906Z shell: bash 2022-11-23T03:20:54.1038314Z timeout_minutes: 10 2022-11-23T03:20:54.1038714Z max_attempts: 5 2022-11-23T03:20:54.1039124Z retry_wait_seconds: 30 2022-11-23T03:20:54.1039982Z command: set -eux python3 -m pip install requests==2.26.0 GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}" 2022-11-23T03:20:54.1040962Z polling_interval_seconds: 1 2022-11-23T03:20:54.1041388Z warning_on_retry: true 2022-11-23T03:20:54.1041831Z continue_on_error: false 2022-11-23T03:20:54.1042241Z env: 2022-11-23T03:20:54.1042636Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:20:54.1043146Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:20:54.1043836Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:20:54.1044623Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:20:54.1046353Z GITHUB_TOKEN: *** 2022-11-23T03:20:54.1047468Z ##[endgroup] 2022-11-23T03:20:54.1784228Z + python3 -m pip install requests==2.26.0 2022-11-23T03:20:55.0047150Z Collecting requests==2.26.0 2022-11-23T03:20:55.1227494Z Using cached https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl 2022-11-23T03:20:55.1417506Z Collecting certifi>=2017.4.17 (from requests==2.26.0) 2022-11-23T03:20:55.1826301Z Using cached https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl 2022-11-23T03:20:55.1878092Z Collecting urllib3<1.27,>=1.21.1 (from requests==2.26.0) 2022-11-23T03:20:55.2478447Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T03:20:55.2664442Z Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests==2.26.0) 2022-11-23T03:20:55.3582649Z Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2022-11-23T03:20:55.3618103Z Collecting idna<4,>=2.5; python_version >= "3" (from requests==2.26.0) 2022-11-23T03:20:55.3879952Z Using cached https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl 2022-11-23T03:20:55.3913233Z Installing collected packages: certifi, urllib3, charset-normalizer, idna, requests 2022-11-23T03:20:55.5799430Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.27.1 urllib3-1.26.12 2022-11-23T03:20:55.6281634Z ++ python3 .github/scripts/get_workflow_job_id.py 3528394938 worker-rocm-amd-106 2022-11-23T03:20:59.7887589Z + GHA_WORKFLOW_JOB_ID=9655437988 2022-11-23T03:20:59.7889006Z + echo job-id=9655437988 2022-11-23T03:21:00.1842422Z Command completed after 1 attempt(s). 2022-11-23T03:21:00.2067106Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2022-11-23T03:21:00.2067491Z kill "$MONITOR_SCRIPT_PID" 2022-11-23T03:21:00.2101008Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T03:21:00.2101309Z env: 2022-11-23T03:21:00.2101575Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:21:00.2101915Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:21:00.2102436Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:21:00.2103034Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:21:00.2103446Z MONITOR_SCRIPT_PID: 41687 2022-11-23T03:21:00.2103705Z ##[endgroup] 2022-11-23T03:21:00.2330693Z Prepare all required actions 2022-11-23T03:21:00.2331062Z Getting action download info 2022-11-23T03:21:00.4874886Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2022-11-23T03:21:01.5291135Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb) 2022-11-23T03:21:02.4528259Z ##[group]Run ./.github/actions/upload-test-artifacts 2022-11-23T03:21:02.4528567Z with: 2022-11-23T03:21:02.4528814Z use-gha: true 2022-11-23T03:21:02.4529183Z file-suffix: test-distributed-2-2-linux.rocm.gpu_9655437988 2022-11-23T03:21:02.4529521Z env: 2022-11-23T03:21:02.4529772Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:21:02.4530080Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:21:02.4530593Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:21:02.4531234Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:21:02.4531806Z ##[endgroup] 2022-11-23T03:21:02.4604485Z ##[group]Run actions/upload-artifact@v3 2022-11-23T03:21:02.4604783Z with: 2022-11-23T03:21:02.4605209Z name: test-jsons-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437988.zip 2022-11-23T03:21:02.4605654Z retention-days: 14 2022-11-23T03:21:02.4605919Z if-no-files-found: warn 2022-11-23T03:21:02.4606214Z path: test/**/*.json 2022-11-23T03:21:02.4606463Z env: 2022-11-23T03:21:02.4606715Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:21:02.4607041Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:21:02.4607553Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:21:02.4608120Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:21:02.4608505Z ##[endgroup] 2022-11-23T03:21:02.7278003Z With the provided path, there will be 3 files uploaded 2022-11-23T03:21:02.7281020Z Starting artifact upload 2022-11-23T03:21:02.7284535Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T03:21:02.7286181Z Artifact name is valid! 2022-11-23T03:21:02.9360461Z Container for artifact "test-jsons-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437988.zip" successfully created. Starting upload of file(s) 2022-11-23T03:21:03.5033665Z Total size of all the files uploaded is 29304 bytes 2022-11-23T03:21:03.5035544Z File upload process has finished. Finalizing the artifact upload 2022-11-23T03:21:03.6906550Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T03:21:03.6907716Z 2022-11-23T03:21:03.6908276Z The raw size of all the files that were specified for upload is 301646 bytes 2022-11-23T03:21:03.6909715Z The size of all the files that were uploaded is 29304 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T03:21:03.6910603Z 2022-11-23T03:21:03.6912183Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T03:21:03.6913245Z 2022-11-23T03:21:03.6914499Z Artifact test-jsons-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437988.zip has been successfully uploaded! 2022-11-23T03:21:03.7077888Z ##[group]Run actions/upload-artifact@v3 2022-11-23T03:21:03.7078591Z with: 2022-11-23T03:21:03.7079661Z name: test-reports-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437988.zip 2022-11-23T03:21:03.7080967Z retention-days: 14 2022-11-23T03:21:03.7081630Z if-no-files-found: ignore 2022-11-23T03:21:03.7082361Z path: test/**/*.xml test/**/*.csv 2022-11-23T03:21:03.7083002Z env: 2022-11-23T03:21:03.7083599Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:21:03.7084670Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:21:03.7085980Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:21:03.7087437Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:21:03.7088394Z ##[endgroup] 2022-11-23T03:21:04.0294576Z With the provided path, there will be 803 files uploaded 2022-11-23T03:21:04.0297773Z Starting artifact upload 2022-11-23T03:21:04.0298768Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T03:21:04.0299403Z Artifact name is valid! 2022-11-23T03:21:04.2615108Z Container for artifact "test-reports-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437988.zip" successfully created. Starting upload of file(s) 2022-11-23T03:21:14.2790963Z Total file count: 803 ---- Processed file #73 (9.0%) 2022-11-23T03:21:24.2790177Z Total file count: 803 ---- Processed file #142 (17.6%) 2022-11-23T03:21:34.2787574Z Total file count: 803 ---- Processed file #219 (27.2%) 2022-11-23T03:21:44.2806766Z Total file count: 803 ---- Processed file #295 (36.7%) 2022-11-23T03:21:54.2816245Z Total file count: 803 ---- Processed file #368 (45.8%) 2022-11-23T03:22:04.2826055Z Total file count: 803 ---- Processed file #443 (55.1%) 2022-11-23T03:22:14.2836336Z Total file count: 803 ---- Processed file #512 (63.7%) 2022-11-23T03:22:24.2838616Z Total file count: 803 ---- Processed file #589 (73.3%) 2022-11-23T03:22:34.2873183Z Total file count: 803 ---- Processed file #661 (82.3%) 2022-11-23T03:22:44.2871714Z Total file count: 803 ---- Processed file #737 (91.7%) 2022-11-23T03:22:53.3920117Z Total size of all the files uploaded is 259103 bytes 2022-11-23T03:22:53.3921494Z File upload process has finished. Finalizing the artifact upload 2022-11-23T03:22:53.5787319Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T03:22:53.5788004Z 2022-11-23T03:22:53.5788524Z The raw size of all the files that were specified for upload is 778512 bytes 2022-11-23T03:22:53.5793921Z The size of all the files that were uploaded is 259103 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T03:22:53.5794857Z 2022-11-23T03:22:53.5796622Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T03:22:53.5797706Z 2022-11-23T03:22:53.5802707Z Artifact test-reports-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437988.zip has been successfully uploaded! 2022-11-23T03:22:53.6034447Z ##[group]Run actions/upload-artifact@v3 2022-11-23T03:22:53.6035162Z with: 2022-11-23T03:22:53.6036229Z name: usage-log-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437988.zip 2022-11-23T03:22:53.6037341Z retention-days: 14 2022-11-23T03:22:53.6038010Z if-no-files-found: ignore 2022-11-23T03:22:53.6038706Z path: usage_log.txt 2022-11-23T03:22:53.6039307Z env: 2022-11-23T03:22:53.6039913Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:22:53.6040949Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:22:53.6042234Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:22:53.6043714Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:22:53.6044690Z ##[endgroup] 2022-11-23T03:22:53.6859295Z With the provided path, there will be 1 file uploaded 2022-11-23T03:22:53.6861493Z Starting artifact upload 2022-11-23T03:22:53.6864610Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T03:22:53.6866280Z Artifact name is valid! 2022-11-23T03:22:54.1192894Z Container for artifact "usage-log-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437988.zip" successfully created. Starting upload of file(s) 2022-11-23T03:22:54.9633202Z Total size of all the files uploaded is 325739 bytes 2022-11-23T03:22:54.9634302Z File upload process has finished. Finalizing the artifact upload 2022-11-23T03:22:55.1598596Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T03:22:55.1599820Z 2022-11-23T03:22:55.1600895Z The raw size of all the files that were specified for upload is 7328098 bytes 2022-11-23T03:22:55.1602544Z The size of all the files that were uploaded is 325739 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T03:22:55.1604244Z 2022-11-23T03:22:55.1606218Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T03:22:55.1607289Z 2022-11-23T03:22:55.1609082Z Artifact usage-log-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437988.zip has been successfully uploaded! 2022-11-23T03:22:55.1752817Z ##[group]Run set -x 2022-11-23T03:22:55.1753426Z set -x 2022-11-23T03:22:55.1754110Z python3 -m pip install -r requirements.txt 2022-11-23T03:22:55.1754933Z python3 -m pip install boto3==1.19.12 2022-11-23T03:22:55.1755892Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T03:22:55.1809988Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T03:22:55.1810310Z env: 2022-11-23T03:22:55.1810574Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:22:55.1810912Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:22:55.1811434Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:22:55.1812007Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:22:55.1812431Z AWS_DEFAULT_REGION: us-east-1 2022-11-23T03:22:55.1812714Z BRANCH: master 2022-11-23T03:22:55.1813034Z TEST_CONFIG: distributed 2022-11-23T03:22:55.1813317Z SHARD_NUMBER: 2 2022-11-23T03:22:55.1813622Z BUILD_ENVIRONMENT: linux-focal-rocm5.2-py3.8 2022-11-23T03:22:55.1813941Z PR_NUMBER: 2022-11-23T03:22:55.1814221Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T03:22:55.1814531Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T03:22:55.1814870Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T03:22:55.1815160Z TAG: 2022-11-23T03:22:55.1815393Z WORKFLOW_ID: 3528394938 2022-11-23T03:22:55.1815877Z GITHUB_TOKEN: *** 2022-11-23T03:22:55.1816207Z AWS_ACCESS_KEY_ID: *** 2022-11-23T03:22:55.1816607Z AWS_SECRET_ACCESS_KEY: *** 2022-11-23T03:22:55.1816903Z GHA_WORKFLOW_JOB_ID: 9655437988 2022-11-23T03:22:55.1817178Z ##[endgroup] 2022-11-23T03:22:55.1878671Z + python3 -m pip install -r requirements.txt 2022-11-23T03:22:56.0109511Z Collecting astunparse (from -r requirements.txt (line 2)) 2022-11-23T03:22:56.0729433Z Using cached https://files.pythonhosted.org/packages/2b/03/13dde6512ad7b4557eb792fbcf0c653af6076b81e5941d36ec61f7ce6028/astunparse-1.6.3-py2.py3-none-any.whl 2022-11-23T03:22:56.0786205Z Collecting expecttest (from -r requirements.txt (line 3)) 2022-11-23T03:22:56.1674458Z Using cached https://files.pythonhosted.org/packages/a6/26/1a287e44618c14659db0256bc1ee239c2134f9c863cb9a85813ecab73413/expecttest-0.1.4-py3-none-any.whl 2022-11-23T03:22:56.1694700Z Collecting future (from -r requirements.txt (line 4)) 2022-11-23T03:22:56.2278004Z Collecting hypothesis (from -r requirements.txt (line 5)) 2022-11-23T03:22:56.8998986Z Using cached https://files.pythonhosted.org/packages/82/7e/01691560a3a98bb1ae909affde49392f599eef3a0b91c27b992bbbde2abb/hypothesis-6.31.6-py3-none-any.whl 2022-11-23T03:22:56.9451052Z Collecting numpy (from -r requirements.txt (line 6)) 2022-11-23T03:22:57.5589556Z Using cached https://files.pythonhosted.org/packages/45/b2/6c7545bb7a38754d63048c7696804a0d947328125d81bf12beaa692c3ae3/numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl 2022-11-23T03:22:57.8784247Z Collecting psutil (from -r requirements.txt (line 7)) 2022-11-23T03:22:58.1039557Z Collecting pyyaml (from -r requirements.txt (line 8)) 2022-11-23T03:22:58.1825945Z Using cached https://files.pythonhosted.org/packages/b3/85/79b9e5b4e8d3c0ac657f4e8617713cca8408f6cdc65d2ee6554217cedff1/PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl 2022-11-23T03:22:58.1966075Z Collecting requests (from -r requirements.txt (line 9)) 2022-11-23T03:22:58.2412524Z Using cached https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl 2022-11-23T03:22:58.2542127Z Collecting setuptools (from -r requirements.txt (line 10)) 2022-11-23T03:22:58.5633776Z Using cached https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl 2022-11-23T03:22:58.6091033Z Collecting six (from -r requirements.txt (line 11)) 2022-11-23T03:22:58.6520643Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-11-23T03:22:58.6536073Z Collecting types-dataclasses (from -r requirements.txt (line 12)) 2022-11-23T03:22:58.7410885Z Using cached https://files.pythonhosted.org/packages/31/85/23ab2bbc280266af5bf22ded4e070946d1694d1721ced90666b649eaa795/types_dataclasses-0.6.6-py3-none-any.whl 2022-11-23T03:22:58.7419889Z Collecting typing_extensions (from -r requirements.txt (line 13)) 2022-11-23T03:22:58.7626865Z Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl 2022-11-23T03:22:58.7643647Z Collecting sympy (from -r requirements.txt (line 14)) 2022-11-23T03:22:58.8602752Z Using cached https://files.pythonhosted.org/packages/78/43/33c5a5e7fbafbf51520f4e09cb0634a1ca1d4cd5469c57967e43183d7a42/sympy-1.9-py3-none-any.whl 2022-11-23T03:22:59.0955754Z Collecting filelock (from -r requirements.txt (line 15)) 2022-11-23T03:22:59.1180343Z Using cached https://files.pythonhosted.org/packages/84/ce/8916d10ef537f3f3b046843255f9799504aa41862bfa87844b9bdc5361cd/filelock-3.4.1-py3-none-any.whl 2022-11-23T03:22:59.1257167Z Collecting networkx (from -r requirements.txt (line 16)) 2022-11-23T03:22:59.1861659Z Using cached https://files.pythonhosted.org/packages/f3/b7/c7f488101c0bb5e4178f3cde416004280fd40262433496830de8a8c21613/networkx-2.5.1-py3-none-any.whl 2022-11-23T03:22:59.2660279Z Collecting jinja2 (from -r requirements.txt (line 17)) 2022-11-23T03:22:59.2962747Z Using cached https://files.pythonhosted.org/packages/20/9a/e5d9ec41927401e41aea8af6d16e78b5e612bca4699d417f646a9610a076/Jinja2-3.0.3-py3-none-any.whl 2022-11-23T03:22:59.3031932Z Collecting wheel<1.0,>=0.23.0 (from astunparse->-r requirements.txt (line 2)) 2022-11-23T03:22:59.3440700Z Using cached https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl 2022-11-23T03:22:59.3482126Z Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis->-r requirements.txt (line 5)) 2022-11-23T03:22:59.3711429Z Using cached https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl 2022-11-23T03:22:59.3735136Z Collecting attrs>=19.2.0 (from hypothesis->-r requirements.txt (line 5)) 2022-11-23T03:22:59.4027584Z Using cached https://files.pythonhosted.org/packages/f2/bc/d817287d1aa01878af07c19505fafd1165cd6a119e9d0821ca1d1c20312d/attrs-22.1.0-py2.py3-none-any.whl 2022-11-23T03:22:59.4267289Z Collecting urllib3<1.27,>=1.21.1 (from requests->-r requirements.txt (line 9)) 2022-11-23T03:22:59.4661910Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T03:22:59.4844587Z Collecting certifi>=2017.4.17 (from requests->-r requirements.txt (line 9)) 2022-11-23T03:22:59.5077594Z Using cached https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl 2022-11-23T03:22:59.5113973Z Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests->-r requirements.txt (line 9)) 2022-11-23T03:22:59.5932551Z Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2022-11-23T03:22:59.5969761Z Collecting idna<4,>=2.5; python_version >= "3" (from requests->-r requirements.txt (line 9)) 2022-11-23T03:22:59.6114452Z Using cached https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl 2022-11-23T03:22:59.6146229Z Collecting mpmath>=0.19 (from sympy->-r requirements.txt (line 14)) 2022-11-23T03:22:59.6574572Z Using cached https://files.pythonhosted.org/packages/d4/cf/3965bddbb4f1a61c49aacae0e78fd1fe36b5dc36c797b31f30cf07dcbbb7/mpmath-1.2.1-py3-none-any.whl 2022-11-23T03:22:59.6785686Z Collecting decorator<5,>=4.3 (from networkx->-r requirements.txt (line 16)) 2022-11-23T03:22:59.7304880Z Using cached https://files.pythonhosted.org/packages/ed/1b/72a1821152d07cf1d8b6fce298aeb06a7eb90f4d6d41acec9861e7cc6df0/decorator-4.4.2-py2.py3-none-any.whl 2022-11-23T03:22:59.7320122Z Collecting MarkupSafe>=2.0 (from jinja2->-r requirements.txt (line 17)) 2022-11-23T03:22:59.8026636Z Using cached https://files.pythonhosted.org/packages/fc/d6/57f9a97e56447a1e340f8574836d3b636e2c14de304943836bd645fa9c7e/MarkupSafe-2.0.1-cp36-cp36m-manylinux1_x86_64.whl 2022-11-23T03:22:59.8053276Z Installing collected packages: wheel, six, astunparse, expecttest, future, sortedcontainers, attrs, hypothesis, numpy, psutil, pyyaml, urllib3, certifi, charset-normalizer, idna, requests, setuptools, types-dataclasses, typing-extensions, mpmath, sympy, filelock, decorator, networkx, MarkupSafe, jinja2 2022-11-23T03:23:08.4105171Z Successfully installed MarkupSafe-2.0.1 astunparse-1.6.3 attrs-22.1.0 certifi-2022.9.24 charset-normalizer-2.0.12 decorator-4.4.2 expecttest-0.1.4 filelock-3.4.1 future-0.18.2 hypothesis-6.31.6 idna-3.4 jinja2-3.0.3 mpmath-1.2.1 networkx-2.5.1 numpy-1.19.5 psutil-5.9.4 pyyaml-6.0 requests-2.27.1 setuptools-59.6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.9 types-dataclasses-0.6.6 typing-extensions-4.1.1 urllib3-1.26.12 wheel-0.37.1 2022-11-23T03:23:08.5542087Z + python3 -m pip install boto3==1.19.12 2022-11-23T03:23:09.3723253Z Collecting boto3==1.19.12 2022-11-23T03:23:09.9072260Z Using cached https://files.pythonhosted.org/packages/5e/e1/156846b09fca21b9b164c54200011e3bd17f29187cbfc6903a8e0281a304/boto3-1.19.12-py3-none-any.whl 2022-11-23T03:23:09.9194109Z Collecting s3transfer<0.6.0,>=0.5.0 (from boto3==1.19.12) 2022-11-23T03:23:09.9432769Z Using cached https://files.pythonhosted.org/packages/7b/9c/f51775ebe7df5a7aa4e7c79ed671bde94e154bd968aca8d65bb24aba0c8c/s3transfer-0.5.2-py3-none-any.whl 2022-11-23T03:23:09.9488045Z Collecting botocore<1.23.0,>=1.22.12 (from boto3==1.19.12) 2022-11-23T03:23:10.7090908Z Using cached https://files.pythonhosted.org/packages/6a/73/552b27e3a1b4f83630907c4958be78e9d4c906e73efd554ebd5e21cb1692/botocore-1.22.12-py3-none-any.whl 2022-11-23T03:23:11.0001068Z Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.19.12) 2022-11-23T03:23:11.0244015Z Using cached https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl 2022-11-23T03:23:11.0271922Z Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T03:23:11.0549881Z Using cached https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl 2022-11-23T03:23:11.0614060Z Collecting urllib3<1.27,>=1.25.4 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T03:23:11.1016646Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T03:23:11.1192138Z Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T03:23:11.1330152Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-11-23T03:23:11.1346984Z Installing collected packages: six, python-dateutil, urllib3, jmespath, botocore, s3transfer, boto3 2022-11-23T03:23:11.6394546Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.12 2022-11-23T03:23:11.7263822Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T03:23:22.2900619Z [scribe] Scribe access token not provided, sending report via boto3... 2022-11-23T03:23:22.2903335Z 2022-11-23T03:23:22.2905120Z ----- Historic stats comparison result ------ 2022-11-23T03:23:22.2905739Z 2022-11-23T03:23:22.2910125Z job: linux-focal-rocm5.2-py3.8 2022-11-23T03:23:22.2911038Z commit: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T03:23:22.2912055Z 2022-11-23T03:23:22.2913211Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-11-23T03:23:22.2913890Z 2022-11-23T03:23:22.2914098Z : (master) 2022-11-23T03:23:22.2914649Z | 2022-11-23T03:23:22.2920896Z * 1cfd3858ac (HEAD) total time 1206.13s 2022-11-23T03:23:22.2929275Z * 26322544b8 (base) 2 reports, total time 10567.84s � 1421.60s 2022-11-23T03:23:22.2930570Z * 7f4b4d2827 2 reports, total time 8461.32s � 1168.97s 2022-11-23T03:23:22.2931732Z * b50699f247 2 reports, total time 9450.96s � 2404.87s 2022-11-23T03:23:22.2932916Z * 8bf8e4d71e 2 reports, total time 9492.36s � 2525.06s 2022-11-23T03:23:22.2934093Z * ce856cee7e 2 reports, total time 10405.71s � 1187.54s 2022-11-23T03:23:22.2935263Z * 391b593ca2 2 reports, total time 10409.50s � 1113.28s 2022-11-23T03:23:22.2936397Z * 5bba783d21 2 reports, total time 10442.70s � 1177.08s 2022-11-23T03:23:22.2937555Z * ea920a1115 2 reports, total time 8384.52s � 1109.09s 2022-11-23T03:23:22.2938713Z * 74e62a1fef 2 reports, total time 8516.95s � 1120.62s 2022-11-23T03:23:22.2939854Z * 00b7d8ef23 2 reports, total time 10408.58s � 1157.84s 2022-11-23T03:23:22.2940547Z | 2022-11-23T03:23:22.2941043Z : 2022-11-23T03:23:22.2941359Z 2022-11-23T03:23:22.2941774Z Removed (across 518 suites) 0 tests, totaling 0.00s 2022-11-23T03:23:22.2942689Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-11-23T03:23:22.2943590Z Added (across 86 suites) 930 tests, totaling +1682.02s 2022-11-23T03:23:22.3434326Z ##[group]Run # Only stop the docker container we started since there might be multiple runners on this host. 2022-11-23T03:23:22.3435778Z # Only stop the docker container we started since there might be multiple runners on this host. 2022-11-23T03:23:22.3437130Z docker stop "ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92" || true 2022-11-23T03:23:22.3438280Z # Prune all of the docker containers. 2022-11-23T03:23:22.3439279Z # Might fail if a prune is already in progress by another runner. 2022-11-23T03:23:22.3440481Z docker container prune -f || true 2022-11-23T03:23:22.3441438Z # Prune everything docker if there are more than 10 images (~200GB). 2022-11-23T03:23:22.3442485Z # This is easier than using a time filter, e.g., "until=24h". 2022-11-23T03:23:22.3443523Z # Might fail if a prune is already in progress by another runner. 2022-11-23T03:23:22.3444527Z image_count=$(docker images | wc -l) 2022-11-23T03:23:22.3445348Z if [[ ${image_count} -gt 10 ]]; then 2022-11-23T03:23:22.3446154Z  echo "Purging all docker caches" 2022-11-23T03:23:22.3446952Z  docker system prune -af || true 2022-11-23T03:23:22.3447665Z else 2022-11-23T03:23:22.3448490Z  echo "Will not purge docker, only ${image_count} images found" 2022-11-23T03:23:22.3449293Z fi 2022-11-23T03:23:22.3515293Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T03:23:22.3516053Z env: 2022-11-23T03:23:22.3516669Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:23:22.3517462Z DOCKER_HOST: unix:///run/user/1120/docker.sock 2022-11-23T03:23:22.3518748Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:23:22.3520365Z CONTAINER_NAME: ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:23:22.3521608Z ##[endgroup] 2022-11-23T03:23:22.8767542Z ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:23:34.2953289Z Deleted Containers: 2022-11-23T03:23:34.2954405Z ff81ba00bb816e5b726fb386ccabaae82f2a1eab95a3f2b4a2ff6e26e0ee1f92 2022-11-23T03:23:34.2955083Z 2022-11-23T03:23:34.2955420Z Total reclaimed space: 8.468GB 2022-11-23T03:23:34.3614691Z Will not purge docker, only 3 images found 2022-11-23T03:23:34.3745590Z Post job cleanup. 2022-11-23T03:23:34.3793868Z Post job cleanup. 2022-11-23T03:23:34.5240410Z [command]/usr/bin/git version 2022-11-23T03:23:34.5304646Z git version 2.37.1 2022-11-23T03:23:34.5360576Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/0289c72a-6f9b-4dcd-b345-157b2d4deeac' before making global git config changes 2022-11-23T03:23:34.5362162Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T03:23:34.5364501Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T03:23:34.5431378Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T03:23:34.5493796Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T03:23:34.5926642Z Entering 'android/libs/fbjni' 2022-11-23T03:23:34.5998840Z Entering 'third_party/FP16' 2022-11-23T03:23:34.6081851Z Entering 'third_party/FXdiv' 2022-11-23T03:23:34.6151076Z Entering 'third_party/NNPACK' 2022-11-23T03:23:34.6229508Z Entering 'third_party/QNNPACK' 2022-11-23T03:23:34.6308378Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T03:23:34.6385816Z Entering 'third_party/XNNPACK' 2022-11-23T03:23:34.6492918Z Entering 'third_party/benchmark' 2022-11-23T03:23:34.6564497Z Entering 'third_party/cpuinfo' 2022-11-23T03:23:34.6637702Z Entering 'third_party/cub' 2022-11-23T03:23:34.6706294Z Entering 'third_party/cudnn_frontend' 2022-11-23T03:23:34.6786102Z Entering 'third_party/cutlass' 2022-11-23T03:23:34.6867473Z Entering 'third_party/eigen' 2022-11-23T03:23:34.6943066Z Entering 'third_party/fbgemm' 2022-11-23T03:23:34.7022331Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T03:23:34.7106119Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T03:23:34.7191984Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T03:23:34.7260542Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T03:23:34.7335674Z Entering 'third_party/flatbuffers' 2022-11-23T03:23:34.7431885Z Entering 'third_party/fmt' 2022-11-23T03:23:34.7521112Z Entering 'third_party/foxi' 2022-11-23T03:23:34.7604392Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T03:23:34.7685538Z Entering 'third_party/gloo' 2022-11-23T03:23:34.7762429Z Entering 'third_party/googletest' 2022-11-23T03:23:34.7841041Z Entering 'third_party/ideep' 2022-11-23T03:23:34.7917012Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T03:23:34.8008123Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T03:23:34.8117169Z Entering 'third_party/ios-cmake' 2022-11-23T03:23:34.8204702Z Entering 'third_party/ittapi' 2022-11-23T03:23:34.8286839Z Entering 'third_party/kineto' 2022-11-23T03:23:34.8363033Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T03:23:34.8444555Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T03:23:34.8536443Z Entering 'third_party/nccl/nccl' 2022-11-23T03:23:34.8625037Z Entering 'third_party/neon2sse' 2022-11-23T03:23:34.8710067Z Entering 'third_party/nlohmann' 2022-11-23T03:23:34.8796990Z Entering 'third_party/onnx' 2022-11-23T03:23:34.8915309Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T03:23:34.9000446Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T03:23:34.9084781Z Entering 'third_party/onnx-tensorrt' 2022-11-23T03:23:34.9171399Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T03:23:34.9266211Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T03:23:34.9351409Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T03:23:34.9437883Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T03:23:34.9537439Z Entering 'third_party/pocketfft' 2022-11-23T03:23:34.9625312Z Entering 'third_party/protobuf' 2022-11-23T03:23:34.9723472Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T03:23:34.9806867Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T03:23:34.9897209Z Entering 'third_party/psimd' 2022-11-23T03:23:34.9984885Z Entering 'third_party/pthreadpool' 2022-11-23T03:23:35.0071722Z Entering 'third_party/pybind11' 2022-11-23T03:23:35.0159858Z Entering 'third_party/python-enum' 2022-11-23T03:23:35.0246489Z Entering 'third_party/python-peachpy' 2022-11-23T03:23:35.0331433Z Entering 'third_party/python-six' 2022-11-23T03:23:35.0415684Z Entering 'third_party/sleef' 2022-11-23T03:23:35.0502535Z Entering 'third_party/tbb' 2022-11-23T03:23:35.0595708Z Entering 'third_party/tensorpipe' 2022-11-23T03:23:35.0675042Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T03:23:35.0760622Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T03:23:35.0843804Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T03:23:35.0929910Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T03:23:35.1013508Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T03:23:35.1106349Z Entering 'third_party/zstd' 2022-11-23T03:23:35.1231571Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T03:23:35.1299161Z http.https://github.com/.extraheader 2022-11-23T03:23:35.1321425Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-11-23T03:23:35.1399999Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T03:23:35.1922496Z Entering 'android/libs/fbjni' 2022-11-23T03:23:35.1967639Z http.https://github.com/.extraheader 2022-11-23T03:23:35.2041925Z Entering 'third_party/FP16' 2022-11-23T03:23:35.2087290Z http.https://github.com/.extraheader 2022-11-23T03:23:35.2159321Z Entering 'third_party/FXdiv' 2022-11-23T03:23:35.2204532Z http.https://github.com/.extraheader 2022-11-23T03:23:35.2273659Z Entering 'third_party/NNPACK' 2022-11-23T03:23:35.2318766Z http.https://github.com/.extraheader 2022-11-23T03:23:35.2394609Z Entering 'third_party/QNNPACK' 2022-11-23T03:23:35.2440466Z http.https://github.com/.extraheader 2022-11-23T03:23:35.2508371Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T03:23:35.2552246Z http.https://github.com/.extraheader 2022-11-23T03:23:35.2627261Z Entering 'third_party/XNNPACK' 2022-11-23T03:23:35.2672293Z http.https://github.com/.extraheader 2022-11-23T03:23:35.2767778Z Entering 'third_party/benchmark' 2022-11-23T03:23:35.2813390Z http.https://github.com/.extraheader 2022-11-23T03:23:35.2885117Z Entering 'third_party/cpuinfo' 2022-11-23T03:23:35.2930817Z http.https://github.com/.extraheader 2022-11-23T03:23:35.3004259Z Entering 'third_party/cub' 2022-11-23T03:23:35.3049237Z http.https://github.com/.extraheader 2022-11-23T03:23:35.3121944Z Entering 'third_party/cudnn_frontend' 2022-11-23T03:23:35.3166981Z http.https://github.com/.extraheader 2022-11-23T03:23:35.3253907Z Entering 'third_party/cutlass' 2022-11-23T03:23:35.3299475Z http.https://github.com/.extraheader 2022-11-23T03:23:35.3386316Z Entering 'third_party/eigen' 2022-11-23T03:23:35.3431323Z http.https://github.com/.extraheader 2022-11-23T03:23:35.3502911Z Entering 'third_party/fbgemm' 2022-11-23T03:23:35.3544384Z http.https://github.com/.extraheader 2022-11-23T03:23:35.3616161Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T03:23:35.3660155Z http.https://github.com/.extraheader 2022-11-23T03:23:35.3732759Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T03:23:35.3775645Z http.https://github.com/.extraheader 2022-11-23T03:23:35.3846385Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T03:23:35.3888145Z http.https://github.com/.extraheader 2022-11-23T03:23:35.3963090Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T03:23:35.4006606Z http.https://github.com/.extraheader 2022-11-23T03:23:35.4087279Z Entering 'third_party/flatbuffers' 2022-11-23T03:23:35.4131064Z http.https://github.com/.extraheader 2022-11-23T03:23:35.4206722Z Entering 'third_party/fmt' 2022-11-23T03:23:35.4251768Z http.https://github.com/.extraheader 2022-11-23T03:23:35.4325342Z Entering 'third_party/foxi' 2022-11-23T03:23:35.4365901Z http.https://github.com/.extraheader 2022-11-23T03:23:35.4436874Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T03:23:35.4482584Z http.https://github.com/.extraheader 2022-11-23T03:23:35.4557358Z Entering 'third_party/gloo' 2022-11-23T03:23:35.4602575Z http.https://github.com/.extraheader 2022-11-23T03:23:35.4673862Z Entering 'third_party/googletest' 2022-11-23T03:23:35.4721068Z http.https://github.com/.extraheader 2022-11-23T03:23:35.4794052Z Entering 'third_party/ideep' 2022-11-23T03:23:35.4837128Z http.https://github.com/.extraheader 2022-11-23T03:23:35.4906686Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T03:23:35.4950249Z http.https://github.com/.extraheader 2022-11-23T03:23:35.5026389Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T03:23:35.5071007Z http.https://github.com/.extraheader 2022-11-23T03:23:35.5164547Z Entering 'third_party/ios-cmake' 2022-11-23T03:23:35.5209512Z http.https://github.com/.extraheader 2022-11-23T03:23:35.5283946Z Entering 'third_party/ittapi' 2022-11-23T03:23:35.5324524Z http.https://github.com/.extraheader 2022-11-23T03:23:35.5399242Z Entering 'third_party/kineto' 2022-11-23T03:23:35.5444757Z http.https://github.com/.extraheader 2022-11-23T03:23:35.5518361Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T03:23:35.5561618Z http.https://github.com/.extraheader 2022-11-23T03:23:35.5630337Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T03:23:35.5670865Z http.https://github.com/.extraheader 2022-11-23T03:23:35.5744724Z Entering 'third_party/nccl/nccl' 2022-11-23T03:23:35.5784744Z http.https://github.com/.extraheader 2022-11-23T03:23:35.5858956Z Entering 'third_party/neon2sse' 2022-11-23T03:23:35.5904556Z http.https://github.com/.extraheader 2022-11-23T03:23:35.5976211Z Entering 'third_party/nlohmann' 2022-11-23T03:23:35.6011741Z http.https://github.com/.extraheader 2022-11-23T03:23:35.6088263Z Entering 'third_party/onnx' 2022-11-23T03:23:35.6129652Z http.https://github.com/.extraheader 2022-11-23T03:23:35.6230826Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T03:23:35.6272640Z http.https://github.com/.extraheader 2022-11-23T03:23:35.6342230Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T03:23:35.6385811Z http.https://github.com/.extraheader 2022-11-23T03:23:35.6466213Z Entering 'third_party/onnx-tensorrt' 2022-11-23T03:23:35.6508874Z http.https://github.com/.extraheader 2022-11-23T03:23:35.6578231Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T03:23:35.6620466Z http.https://github.com/.extraheader 2022-11-23T03:23:35.6707362Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T03:23:35.6750696Z http.https://github.com/.extraheader 2022-11-23T03:23:35.6827055Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T03:23:35.6871682Z http.https://github.com/.extraheader 2022-11-23T03:23:35.6946214Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T03:23:35.6992274Z http.https://github.com/.extraheader 2022-11-23T03:23:35.7085611Z Entering 'third_party/pocketfft' 2022-11-23T03:23:35.7132077Z http.https://github.com/.extraheader 2022-11-23T03:23:35.7205907Z Entering 'third_party/protobuf' 2022-11-23T03:23:35.7252218Z http.https://github.com/.extraheader 2022-11-23T03:23:35.7332151Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T03:23:35.7376354Z http.https://github.com/.extraheader 2022-11-23T03:23:35.7450100Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T03:23:35.7495926Z http.https://github.com/.extraheader 2022-11-23T03:23:35.7574662Z Entering 'third_party/psimd' 2022-11-23T03:23:35.7620698Z http.https://github.com/.extraheader 2022-11-23T03:23:35.7695675Z Entering 'third_party/pthreadpool' 2022-11-23T03:23:35.7741145Z http.https://github.com/.extraheader 2022-11-23T03:23:35.7816769Z Entering 'third_party/pybind11' 2022-11-23T03:23:35.7861816Z http.https://github.com/.extraheader 2022-11-23T03:23:35.7935870Z Entering 'third_party/python-enum' 2022-11-23T03:23:35.7979303Z http.https://github.com/.extraheader 2022-11-23T03:23:35.8054700Z Entering 'third_party/python-peachpy' 2022-11-23T03:23:35.8099826Z http.https://github.com/.extraheader 2022-11-23T03:23:35.8175438Z Entering 'third_party/python-six' 2022-11-23T03:23:35.8220979Z http.https://github.com/.extraheader 2022-11-23T03:23:35.8295757Z Entering 'third_party/sleef' 2022-11-23T03:23:35.8341256Z http.https://github.com/.extraheader 2022-11-23T03:23:35.8413865Z Entering 'third_party/tbb' 2022-11-23T03:23:35.8456048Z http.https://github.com/.extraheader 2022-11-23T03:23:35.8531996Z Entering 'third_party/tensorpipe' 2022-11-23T03:23:35.8577787Z http.https://github.com/.extraheader 2022-11-23T03:23:35.8652894Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T03:23:35.8696556Z http.https://github.com/.extraheader 2022-11-23T03:23:35.8771226Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T03:23:35.8814073Z http.https://github.com/.extraheader 2022-11-23T03:23:35.8885821Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T03:23:35.8928562Z http.https://github.com/.extraheader 2022-11-23T03:23:35.9004245Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T03:23:35.9047893Z http.https://github.com/.extraheader 2022-11-23T03:23:35.9115725Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T03:23:35.9159917Z http.https://github.com/.extraheader 2022-11-23T03:23:35.9243831Z Entering 'third_party/zstd' 2022-11-23T03:23:35.9289134Z http.https://github.com/.extraheader 2022-11-23T03:23:35.9799372Z Cleaning up orphan processes